2023-10-11T10:28:18.697 INFO:root:teuthology version: 0.0.1.dev194+g54e62bc 2023-10-11T10:28:18.698 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --owner scheduled_dis@teuthology --name dis-2023-10-11_10:22:06-rbd-wip-dis-testing-distro-default-smithi --description rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-stupid policy/none rbd-mirror/four-per-cluster supported-random-distro$/{rhel_8} workloads/rbd-mirror-snapshot-stress-workunit-fast-diff} --archive /home/teuthworker/archive/dis-2023-10-11_10:22:06-rbd-wip-dis-testing-distro-default-smithi/7421102 -- /home/teuthworker/archive/dis-2023-10-11_10:22:06-rbd-wip-dis-testing-distro-default-smithi/7421102/orig.config.yaml 2023-10-11T10:28:18.745 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-10-11T10:28:18.832 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/dis-2023-10-11_10:22:06-rbd-wip-dis-testing-distro-default-smithi/7421102 branch: wip-dis-testing description: rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-stupid policy/none rbd-mirror/four-per-cluster supported-random-distro$/{rhel_8} workloads/rbd-mirror-snapshot-stress-workunit-fast-diff} email: idryomov@gmail.com first_in_suite: false job_id: '7421102' kernel: kdb: true sha1: distro ktype: distro last_in_suite: false machine_type: smithi meta: - desc: run two ceph clusters and install rbd-mirror - desc: configure the permissions for client.mirror - desc: 2 ceph clusters with 1 mon and 3 osds each - desc: run four rbd-mirror daemons per cluster - desc: run the rbd_mirror_stress.sh workunit to test the rbd-mirror daemon name: dis-2023-10-11_10:22:06-rbd-wip-dis-testing-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 3 size: 30 os_type: rhel os_version: '8.6' overrides: admin_socket: branch: wip-dis-testing ceph: conf: client: debug rbd: 20 debug rbd_mirror: 15 log to stderr: false rbd default features: 125 rbd mirror image policy type: none client.mirror.0: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.1: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.2: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.3: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid global: mon client directed command retry: 5 mon warn on pool no app: false ms inject socket failures: 5000 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: bdev async discard: true bdev enable discard: true bluestore allocator: stupid bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME sha1: 4bf94f59187ed7eba271559c0898cec750ad93d5 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 4bf94f59187ed7eba271559c0898cec750ad93d5 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-dis-testing-qa sha1: 3d099a3859cc211cf8dd2acdaf3194ba0e96ea9c owner: scheduled_dis@teuthology priority: 74 repo: https://github.com/ceph/ceph-ci roles: - - cluster1.mon.a - cluster1.mgr.x - cluster2.mgr.x - cluster1.osd.0 - cluster1.osd.1 - cluster1.osd.2 - cluster1.client.0 - cluster2.client.0 - - cluster2.mon.a - cluster2.osd.0 - cluster2.osd.1 - cluster2.osd.2 - cluster1.client.mirror - cluster1.client.mirror.0 - cluster1.client.mirror.1 - cluster1.client.mirror.2 - cluster1.client.mirror.3 - cluster1.client.mirror.4 - cluster1.client.mirror.5 - cluster1.client.mirror.6 - cluster2.client.mirror - cluster2.client.mirror.0 - cluster2.client.mirror.1 - cluster2.client.mirror.2 - cluster2.client.mirror.3 - cluster2.client.mirror.4 - cluster2.client.mirror.5 - cluster2.client.mirror.6 seed: 2449 sha1: 4bf94f59187ed7eba271559c0898cec750ad93d5 sleep_before_teardown: 0 subset: 49/99 suite: rbd suite_branch: wip-dis-testing-qa suite_path: /home/teuthworker/src/github.com_idryomov_ceph_3d099a3859cc211cf8dd2acdaf3194ba0e96ea9c/qa suite_relpath: qa suite_repo: https://github.com/idryomov/ceph suite_sha1: 3d099a3859cc211cf8dd2acdaf3194ba0e96ea9c targets: smithi050.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs= smithi111.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs= tasks: - install: extra_packages: - rbd-mirror - ceph: cluster: cluster1 - ceph: cluster: cluster2 - exec: cluster1.client.mirror.0: - sudo ceph --cluster cluster1 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd' - rbd-mirror: client: cluster1.client.mirror.0 thrash: true - rbd-mirror: client: cluster1.client.mirror.1 thrash: true - rbd-mirror: client: cluster1.client.mirror.2 thrash: true - rbd-mirror: client: cluster1.client.mirror.3 thrash: true - rbd-mirror: client: cluster2.client.mirror.0 thrash: true - rbd-mirror: client: cluster2.client.mirror.1 thrash: true - rbd-mirror: client: cluster2.client.mirror.2 thrash: true - rbd-mirror: client: cluster2.client.mirror.3 thrash: true - rbd-mirror-thrash: cluster: cluster1 - rbd-mirror-thrash: cluster: cluster2 - workunit: clients: cluster1.client.mirror: - rbd/rbd_mirror_stress.sh env: CEPH_ARGS: '' MIRROR_IMAGE_MODE: snapshot MIRROR_POOL_MODE: image RBD_IMAGE_FEATURES: layering,exclusive-lock,object-map,fast-diff RBD_MIRROR_INSTANCES: '4' RBD_MIRROR_USE_EXISTING_CLUSTER: '1' RBD_MIRROR_USE_RBD_MIRROR: '1' timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 54e62bcbac4e53d9685e08328b790d3b20d71cae timestamp: 2023-10-11_10:22:06 tube: smithi user: dis verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2830 2023-10-11T10:28:18.832 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_idryomov_ceph_3d099a3859cc211cf8dd2acdaf3194ba0e96ea9c/qa; will attempt to use it 2023-10-11T10:28:18.833 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_idryomov_ceph_3d099a3859cc211cf8dd2acdaf3194ba0e96ea9c/qa/tasks 2023-10-11T10:28:18.833 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-10-11T10:28:18.835 INFO:teuthology.task.internal:Checking packages... 2023-10-11T10:28:18.855 INFO:teuthology.task.internal:Checking packages for os_type 'rhel', flavor 'default' and ceph hash '4bf94f59187ed7eba271559c0898cec750ad93d5' 2023-10-11T10:28:18.856 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-10-11T10:28:18.856 INFO:teuthology.packaging:ref: None 2023-10-11T10:28:18.856 INFO:teuthology.packaging:tag: None 2023-10-11T10:28:18.857 INFO:teuthology.packaging:branch: wip-dis-testing 2023-10-11T10:28:18.857 INFO:teuthology.packaging:sha1: 4bf94f59187ed7eba271559c0898cec750ad93d5 2023-10-11T10:28:18.857 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-dis-testing 2023-10-11T10:28:19.003 INFO:teuthology.task.internal:Found packages for ceph version 18.0.0-6661.g4bf94f59 2023-10-11T10:28:19.003 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-10-11T10:28:19.011 INFO:teuthology.task.internal:no buildpackages task found 2023-10-11T10:28:19.011 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-10-11T10:28:19.090 INFO:teuthology.task.internal:Saving configuration 2023-10-11T10:28:19.107 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-10-11T10:28:19.113 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-11T10:28:19.137 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi050.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/dis-2023-10-11_10:22:06-rbd-wip-dis-testing-distro-default-smithi/7421102', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-11 10:22:14.535993', 'locked_by': 'scheduled_dis@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='} 2023-10-11T10:28:19.161 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi111.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/dis-2023-10-11_10:22:06-rbd-wip-dis-testing-distro-default-smithi/7421102', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-11 10:22:14.537584', 'locked_by': 'scheduled_dis@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='} 2023-10-11T10:28:19.161 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-10-11T10:28:19.167 INFO:teuthology.task.internal:roles: ubuntu@smithi050.front.sepia.ceph.com - ['cluster1.mon.a', 'cluster1.mgr.x', 'cluster2.mgr.x', 'cluster1.osd.0', 'cluster1.osd.1', 'cluster1.osd.2', 'cluster1.client.0', 'cluster2.client.0'] 2023-10-11T10:28:19.168 INFO:teuthology.task.internal:roles: ubuntu@smithi111.front.sepia.ceph.com - ['cluster2.mon.a', 'cluster2.osd.0', 'cluster2.osd.1', 'cluster2.osd.2', 'cluster1.client.mirror', 'cluster1.client.mirror.0', 'cluster1.client.mirror.1', 'cluster1.client.mirror.2', 'cluster1.client.mirror.3', 'cluster1.client.mirror.4', 'cluster1.client.mirror.5', 'cluster1.client.mirror.6', 'cluster2.client.mirror', 'cluster2.client.mirror.0', 'cluster2.client.mirror.1', 'cluster2.client.mirror.2', 'cluster2.client.mirror.3', 'cluster2.client.mirror.4', 'cluster2.client.mirror.5', 'cluster2.client.mirror.6'] 2023-10-11T10:28:19.168 INFO:teuthology.run_tasks:Running task console_log... 2023-10-11T10:28:19.243 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f6b528849d0>, signals=[15]) 2023-10-11T10:28:19.244 INFO:teuthology.run_tasks:Running task internal.connect... 2023-10-11T10:28:19.253 INFO:teuthology.task.internal:Opening connections... 2023-10-11T10:28:19.253 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi050.front.sepia.ceph.com 2023-10-11T10:28:19.256 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi050.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-11T10:28:19.325 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi111.front.sepia.ceph.com 2023-10-11T10:28:19.326 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi111.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-11T10:28:19.391 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-10-11T10:28:19.398 DEBUG:teuthology.orchestra.run.smithi050:> uname -m 2023-10-11T10:28:19.420 INFO:teuthology.orchestra.run.smithi050.stdout:x86_64 2023-10-11T10:28:19.420 DEBUG:teuthology.orchestra.run.smithi050:> cat /etc/os-release 2023-10-11T10:28:19.480 INFO:teuthology.orchestra.run.smithi050.stdout:NAME="Red Hat Enterprise Linux" 2023-10-11T10:28:19.480 INFO:teuthology.orchestra.run.smithi050.stdout:VERSION="8.6 (Ootpa)" 2023-10-11T10:28:19.480 INFO:teuthology.orchestra.run.smithi050.stdout:ID="rhel" 2023-10-11T10:28:19.481 INFO:teuthology.orchestra.run.smithi050.stdout:ID_LIKE="fedora" 2023-10-11T10:28:19.481 INFO:teuthology.orchestra.run.smithi050.stdout:VERSION_ID="8.6" 2023-10-11T10:28:19.481 INFO:teuthology.orchestra.run.smithi050.stdout:PLATFORM_ID="platform:el8" 2023-10-11T10:28:19.481 INFO:teuthology.orchestra.run.smithi050.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2023-10-11T10:28:19.481 INFO:teuthology.orchestra.run.smithi050.stdout:ANSI_COLOR="0;31" 2023-10-11T10:28:19.482 INFO:teuthology.orchestra.run.smithi050.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2023-10-11T10:28:19.482 INFO:teuthology.orchestra.run.smithi050.stdout:HOME_URL="https://www.redhat.com/" 2023-10-11T10:28:19.482 INFO:teuthology.orchestra.run.smithi050.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-10-11T10:28:19.482 INFO:teuthology.orchestra.run.smithi050.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-10-11T10:28:19.482 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-11T10:28:19.483 INFO:teuthology.orchestra.run.smithi050.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-10-11T10:28:19.483 INFO:teuthology.orchestra.run.smithi050.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2023-10-11T10:28:19.483 INFO:teuthology.orchestra.run.smithi050.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-10-11T10:28:19.483 INFO:teuthology.orchestra.run.smithi050.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2023-10-11T10:28:19.484 INFO:teuthology.lock.ops:Updating smithi050.front.sepia.ceph.com on lock server 2023-10-11T10:28:19.505 DEBUG:teuthology.orchestra.run.smithi111:> uname -m 2023-10-11T10:28:19.525 INFO:teuthology.orchestra.run.smithi111.stdout:x86_64 2023-10-11T10:28:19.525 DEBUG:teuthology.orchestra.run.smithi111:> cat /etc/os-release 2023-10-11T10:28:19.584 INFO:teuthology.orchestra.run.smithi111.stdout:NAME="Red Hat Enterprise Linux" 2023-10-11T10:28:19.584 INFO:teuthology.orchestra.run.smithi111.stdout:VERSION="8.6 (Ootpa)" 2023-10-11T10:28:19.584 INFO:teuthology.orchestra.run.smithi111.stdout:ID="rhel" 2023-10-11T10:28:19.585 INFO:teuthology.orchestra.run.smithi111.stdout:ID_LIKE="fedora" 2023-10-11T10:28:19.585 INFO:teuthology.orchestra.run.smithi111.stdout:VERSION_ID="8.6" 2023-10-11T10:28:19.585 INFO:teuthology.orchestra.run.smithi111.stdout:PLATFORM_ID="platform:el8" 2023-10-11T10:28:19.585 INFO:teuthology.orchestra.run.smithi111.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2023-10-11T10:28:19.585 INFO:teuthology.orchestra.run.smithi111.stdout:ANSI_COLOR="0;31" 2023-10-11T10:28:19.585 INFO:teuthology.orchestra.run.smithi111.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2023-10-11T10:28:19.586 INFO:teuthology.orchestra.run.smithi111.stdout:HOME_URL="https://www.redhat.com/" 2023-10-11T10:28:19.586 INFO:teuthology.orchestra.run.smithi111.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-10-11T10:28:19.586 INFO:teuthology.orchestra.run.smithi111.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-10-11T10:28:19.586 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T10:28:19.586 INFO:teuthology.orchestra.run.smithi111.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-10-11T10:28:19.587 INFO:teuthology.orchestra.run.smithi111.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2023-10-11T10:28:19.587 INFO:teuthology.orchestra.run.smithi111.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-10-11T10:28:19.587 INFO:teuthology.orchestra.run.smithi111.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2023-10-11T10:28:19.587 INFO:teuthology.lock.ops:Updating smithi111.front.sepia.ceph.com on lock server 2023-10-11T10:28:19.604 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-10-11T10:28:19.613 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-10-11T10:28:19.634 INFO:teuthology.task.internal:Checking for old test directory... 2023-10-11T10:28:19.635 DEBUG:teuthology.orchestra.run.smithi050:> test '!' -e /home/ubuntu/cephtest 2023-10-11T10:28:19.637 DEBUG:teuthology.orchestra.run.smithi111:> test '!' -e /home/ubuntu/cephtest 2023-10-11T10:28:19.657 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-10-11T10:28:19.664 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-10-11T10:28:19.664 DEBUG:teuthology.orchestra.run.smithi050:> test -z $(ls -A /var/lib/ceph) 2023-10-11T10:28:19.697 DEBUG:teuthology.orchestra.run.smithi111:> test -z $(ls -A /var/lib/ceph) 2023-10-11T10:28:19.740 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-10-11T10:28:19.795 INFO:teuthology.run_tasks:Running task kernel... 2023-10-11T10:28:19.809 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-10-11T10:28:19.810 INFO:teuthology.task.kernel:config {'cluster1.mon.a': {'kdb': True, 'sha1': 'distro'}, 'cluster2.mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-10-11T10:28:19.810 DEBUG:teuthology.orchestra.run.smithi050:> test -f /run/.containerenv -o -f /.dockerenv 2023-10-11T10:28:19.811 DEBUG:teuthology.orchestra.run.smithi111:> test -f /run/.containerenv -o -f /.dockerenv 2023-10-11T10:28:19.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-11T10:28:19.830 DEBUG:teuthology.orchestra.run.smithi111:> uname -r 2023-10-11T10:28:19.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-11T10:28:19.831 DEBUG:teuthology.orchestra.run.smithi050:> uname -r 2023-10-11T10:28:19.888 INFO:teuthology.orchestra.run.smithi111.stdout:4.18.0-372.9.1.el8.x86_64 2023-10-11T10:28:19.888 INFO:teuthology.task.kernel:Running kernel on smithi111: 4.18.0-372.9.1.el8.x86_64 2023-10-11T10:28:19.888 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum install -y kernel 2023-10-11T10:28:19.892 INFO:teuthology.orchestra.run.smithi050.stdout:4.18.0-372.9.1.el8.x86_64 2023-10-11T10:28:19.892 INFO:teuthology.task.kernel:Running kernel on smithi050: 4.18.0-372.9.1.el8.x86_64 2023-10-11T10:28:19.892 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum install -y kernel 2023-10-11T10:28:21.156 INFO:teuthology.orchestra.run.smithi111.stdout:Updating Subscription Management repositories. 2023-10-11T10:28:21.156 INFO:teuthology.orchestra.run.smithi111.stdout:Unable to read consumer identity 2023-10-11T10:28:21.213 INFO:teuthology.orchestra.run.smithi050.stdout:Updating Subscription Management repositories. 2023-10-11T10:28:21.213 INFO:teuthology.orchestra.run.smithi050.stdout:Unable to read consumer identity 2023-10-11T10:28:22.547 INFO:teuthology.orchestra.run.smithi050.stdout:Copr repo for python3-asyncssh owned by ceph 25 kB/s | 3.5 kB 00:00 2023-10-11T10:28:22.599 INFO:teuthology.orchestra.run.smithi111.stdout:Copr repo for python3-asyncssh owned by ceph 22 kB/s | 3.5 kB 00:00 2023-10-11T10:28:23.231 INFO:teuthology.orchestra.run.smithi050.stdout:Extra Packages for Enterprise Linux 24 MB/s | 16 MB 00:00 2023-10-11T10:28:23.833 INFO:teuthology.orchestra.run.smithi111.stdout:Extra Packages for Enterprise Linux 12 MB/s | 16 MB 00:01 2023-10-11T10:28:26.973 INFO:teuthology.orchestra.run.smithi050.stdout:lab-extras 359 kB/s | 24 kB 00:00 2023-10-11T10:28:27.358 INFO:teuthology.orchestra.run.smithi111.stdout:lab-extras 400 kB/s | 24 kB 00:00 2023-10-11T10:28:27.634 INFO:teuthology.orchestra.run.smithi050.stdout:Package kernel-4.18.0-372.9.1.el8.x86_64 is already installed. 2023-10-11T10:28:27.681 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2023-10-11T10:28:27.682 INFO:teuthology.orchestra.run.smithi050.stdout:Nothing to do. 2023-10-11T10:28:27.682 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2023-10-11T10:28:27.810 DEBUG:teuthology.orchestra.run.smithi050:> echo no | sudo yum reinstall kernel || true 2023-10-11T10:28:28.061 INFO:teuthology.orchestra.run.smithi111.stdout:Package kernel-4.18.0-372.9.1.el8.x86_64 is already installed. 2023-10-11T10:28:28.139 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2023-10-11T10:28:28.141 INFO:teuthology.orchestra.run.smithi111.stdout:Nothing to do. 2023-10-11T10:28:28.141 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2023-10-11T10:28:28.165 INFO:teuthology.orchestra.run.smithi050.stdout:Updating Subscription Management repositories. 2023-10-11T10:28:28.165 INFO:teuthology.orchestra.run.smithi050.stdout:Unable to read consumer identity 2023-10-11T10:28:28.250 DEBUG:teuthology.orchestra.run.smithi111:> echo no | sudo yum reinstall kernel || true 2023-10-11T10:28:28.434 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:00:02 ago on Wed 11 Oct 2023 10:28:26 AM UTC. 2023-10-11T10:28:28.478 INFO:teuthology.orchestra.run.smithi050.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2023-10-11T10:28:28.517 INFO:teuthology.orchestra.run.smithi050.stderr:Error: No packages marked for reinstall. 2023-10-11T10:28:28.564 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum reinstall -y kernel || true 2023-10-11T10:28:28.619 INFO:teuthology.orchestra.run.smithi111.stdout:Updating Subscription Management repositories. 2023-10-11T10:28:28.620 INFO:teuthology.orchestra.run.smithi111.stdout:Unable to read consumer identity 2023-10-11T10:28:28.877 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:00:01 ago on Wed 11 Oct 2023 10:28:27 AM UTC. 2023-10-11T10:28:28.913 INFO:teuthology.orchestra.run.smithi050.stdout:Updating Subscription Management repositories. 2023-10-11T10:28:28.913 INFO:teuthology.orchestra.run.smithi050.stdout:Unable to read consumer identity 2023-10-11T10:28:28.921 INFO:teuthology.orchestra.run.smithi111.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2023-10-11T10:28:28.958 INFO:teuthology.orchestra.run.smithi111.stderr:Error: No packages marked for reinstall. 2023-10-11T10:28:29.003 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum reinstall -y kernel || true 2023-10-11T10:28:29.187 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:00:03 ago on Wed 11 Oct 2023 10:28:26 AM UTC. 2023-10-11T10:28:29.235 INFO:teuthology.orchestra.run.smithi050.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2023-10-11T10:28:29.266 INFO:teuthology.orchestra.run.smithi050.stderr:Error: No packages marked for reinstall. 2023-10-11T10:28:29.313 DEBUG:teuthology.orchestra.run.smithi050:> rpm -q kernel | sort -rV | head -n 1 2023-10-11T10:28:29.330 INFO:teuthology.orchestra.run.smithi111.stdout:Updating Subscription Management repositories. 2023-10-11T10:28:29.330 INFO:teuthology.orchestra.run.smithi111.stdout:Unable to read consumer identity 2023-10-11T10:28:29.396 INFO:teuthology.orchestra.run.smithi050.stdout:kernel-4.18.0-372.9.1.el8.x86_64 2023-10-11T10:28:29.396 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-372.9.1.el8.x86_64 2023-10-11T10:28:29.397 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2023-10-11T10:28:29.397 INFO:teuthology.task.kernel:Enabling kdb on cluster1.mon.a... 2023-10-11T10:28:29.397 DEBUG:teuthology.orchestra.run.smithi050:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-10-11T10:28:29.424 INFO:teuthology.orchestra.run.smithi050.stdout:ttyS1 2023-10-11T10:28:29.438 DEBUG:teuthology.parallel:result is None 2023-10-11T10:28:29.580 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:00:02 ago on Wed 11 Oct 2023 10:28:27 AM UTC. 2023-10-11T10:28:29.620 INFO:teuthology.orchestra.run.smithi111.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2023-10-11T10:28:29.655 INFO:teuthology.orchestra.run.smithi111.stderr:Error: No packages marked for reinstall. 2023-10-11T10:28:29.703 DEBUG:teuthology.orchestra.run.smithi111:> rpm -q kernel | sort -rV | head -n 1 2023-10-11T10:28:29.783 INFO:teuthology.orchestra.run.smithi111.stdout:kernel-4.18.0-372.9.1.el8.x86_64 2023-10-11T10:28:29.783 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-372.9.1.el8.x86_64 2023-10-11T10:28:29.784 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2023-10-11T10:28:29.784 INFO:teuthology.task.kernel:Enabling kdb on cluster2.mon.a... 2023-10-11T10:28:29.784 DEBUG:teuthology.orchestra.run.smithi111:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-10-11T10:28:29.810 INFO:teuthology.orchestra.run.smithi111.stdout:ttyS1 2023-10-11T10:28:29.825 DEBUG:teuthology.parallel:result is None 2023-10-11T10:28:29.825 INFO:teuthology.run_tasks:Running task internal.base... 2023-10-11T10:28:29.833 INFO:teuthology.task.internal:Creating test directory... 2023-10-11T10:28:29.834 DEBUG:teuthology.orchestra.run.smithi050:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-10-11T10:28:29.836 DEBUG:teuthology.orchestra.run.smithi111:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-10-11T10:28:29.882 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-10-11T10:28:29.889 INFO:teuthology.run_tasks:Running task internal.archive... 2023-10-11T10:28:29.894 INFO:teuthology.task.internal:Creating archive directory... 2023-10-11T10:28:29.894 DEBUG:teuthology.orchestra.run.smithi050:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-10-11T10:28:29.898 DEBUG:teuthology.orchestra.run.smithi111:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-10-11T10:28:29.961 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-10-11T10:28:29.969 INFO:teuthology.task.internal:Enabling coredump saving... 2023-10-11T10:28:29.969 DEBUG:teuthology.orchestra.run.smithi050:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-10-11T10:28:29.972 DEBUG:teuthology.orchestra.run.smithi111:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-10-11T10:28:30.010 INFO:teuthology.orchestra.run.smithi050.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-11T10:28:30.021 INFO:teuthology.orchestra.run.smithi050.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-11T10:28:30.041 INFO:teuthology.orchestra.run.smithi111.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-11T10:28:30.051 INFO:teuthology.orchestra.run.smithi111.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-11T10:28:30.053 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-10-11T10:28:30.060 INFO:teuthology.task.internal:Configuring sudo... 2023-10-11T10:28:30.060 DEBUG:teuthology.orchestra.run.smithi050:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-10-11T10:28:30.065 DEBUG:teuthology.orchestra.run.smithi111:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-10-11T10:28:30.092 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-10-11T10:28:30.100 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-10-11T10:28:30.101 DEBUG:teuthology.orchestra.run.smithi050:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-10-11T10:28:30.134 DEBUG:teuthology.orchestra.run.smithi111:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-10-11T10:28:30.150 DEBUG:teuthology.orchestra.run.smithi050:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-11T10:28:30.233 DEBUG:teuthology.orchestra.run.smithi050:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-11T10:28:30.285 DEBUG:teuthology.orchestra.run.smithi050:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-11T10:28:30.371 DEBUG:teuthology.orchestra.run.smithi050:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-11T10:28:30.411 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2023-10-11T10:28:30.412 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-10-11T10:28:30.479 DEBUG:teuthology.orchestra.run.smithi111:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-11T10:28:30.521 DEBUG:teuthology.orchestra.run.smithi111:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-11T10:28:30.579 DEBUG:teuthology.orchestra.run.smithi111:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-11T10:28:30.664 DEBUG:teuthology.orchestra.run.smithi111:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-11T10:28:30.705 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2023-10-11T10:28:30.706 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-10-11T10:28:30.780 DEBUG:teuthology.orchestra.run.smithi050:> sudo service rsyslog restart 2023-10-11T10:28:30.782 DEBUG:teuthology.orchestra.run.smithi111:> sudo service rsyslog restart 2023-10-11T10:28:30.836 INFO:teuthology.orchestra.run.smithi050.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-11T10:28:30.875 INFO:teuthology.orchestra.run.smithi111.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-11T10:28:31.353 INFO:teuthology.run_tasks:Running task internal.timer... 2023-10-11T10:28:31.361 INFO:teuthology.task.internal:Starting timer... 2023-10-11T10:28:31.362 INFO:teuthology.run_tasks:Running task pcp... 2023-10-11T10:28:31.373 INFO:teuthology.run_tasks:Running task selinux... 2023-10-11T10:28:31.382 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2023-10-11T10:28:31.382 DEBUG:teuthology.orchestra.run.smithi050:> sudo service auditd rotate 2023-10-11T10:28:31.451 INFO:teuthology.orchestra.run.smithi050.stdout:Rotating logs: 2023-10-11T10:28:31.453 DEBUG:teuthology.orchestra.run.smithi111:> sudo service auditd rotate 2023-10-11T10:28:31.518 INFO:teuthology.orchestra.run.smithi111.stdout:Rotating logs: 2023-10-11T10:28:31.520 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-10-11T10:28:31.520 DEBUG:teuthology.orchestra.run.smithi050:> /usr/sbin/getenforce 2023-10-11T10:28:31.543 INFO:teuthology.orchestra.run.smithi050.stdout:Permissive 2023-10-11T10:28:31.544 DEBUG:teuthology.orchestra.run.smithi111:> /usr/sbin/getenforce 2023-10-11T10:28:31.567 INFO:teuthology.orchestra.run.smithi111.stdout:Permissive 2023-10-11T10:28:31.568 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi050.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi111.front.sepia.ceph.com': 'permissive'} 2023-10-11T10:28:31.568 DEBUG:teuthology.orchestra.run.smithi050:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-11T10:28:31.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-11T10:28:31.613 DEBUG:teuthology.orchestra.run.smithi111:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-11T10:28:31.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-11T10:28:31.642 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-10-11T10:28:31.642 DEBUG:teuthology.orchestra.run.smithi050:> sudo /usr/sbin/setenforce permissive 2023-10-11T10:28:31.682 DEBUG:teuthology.orchestra.run.smithi111:> sudo /usr/sbin/setenforce permissive 2023-10-11T10:28:31.709 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-10-11T10:28:31.718 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main was just updated or references a specific commit; assuming it is current 2023-10-11T10:28:31.718 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-10-11T10:28:31.733 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-10-11T10:28:31.737 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi050.front.sepia.ceph.com,smithi111.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-10-11T10:37:22.887 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi050.front.sepia.ceph.com'), Remote(name='ubuntu@smithi111.front.sepia.ceph.com')] 2023-10-11T10:37:22.889 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi050.front.sepia.ceph.com' 2023-10-11T10:37:22.890 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi050.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-11T10:37:22.959 DEBUG:teuthology.orchestra.run.smithi050:> true 2023-10-11T10:37:23.029 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi050.front.sepia.ceph.com' 2023-10-11T10:37:23.029 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi111.front.sepia.ceph.com' 2023-10-11T10:37:23.030 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi111.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-11T10:37:23.099 DEBUG:teuthology.orchestra.run.smithi111:> true 2023-10-11T10:37:23.123 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi111.front.sepia.ceph.com' 2023-10-11T10:37:23.123 INFO:teuthology.run_tasks:Running task clock... 2023-10-11T10:37:23.134 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-10-11T10:37:23.135 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-10-11T10:37:23.135 DEBUG:teuthology.orchestra.run.smithi050:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-11T10:37:23.137 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-10-11T10:37:23.138 DEBUG:teuthology.orchestra.run.smithi111:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-11T10:37:23.167 INFO:teuthology.orchestra.run.smithi050.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-10-11T10:37:23.183 INFO:teuthology.orchestra.run.smithi050.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-10-11T10:37:23.194 INFO:teuthology.orchestra.run.smithi111.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-10-11T10:37:23.208 INFO:teuthology.orchestra.run.smithi111.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-10-11T10:37:23.245 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: ntpd: command not found 2023-10-11T10:37:23.257 INFO:teuthology.orchestra.run.smithi050.stdout:506 Cannot talk to daemon 2023-10-11T10:37:23.272 INFO:teuthology.orchestra.run.smithi050.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-10-11T10:37:23.286 INFO:teuthology.orchestra.run.smithi050.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-10-11T10:37:23.288 INFO:teuthology.orchestra.run.smithi111.stderr:sudo: ntpd: command not found 2023-10-11T10:37:23.299 INFO:teuthology.orchestra.run.smithi111.stdout:506 Cannot talk to daemon 2023-10-11T10:37:23.314 INFO:teuthology.orchestra.run.smithi111.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-10-11T10:37:23.327 INFO:teuthology.orchestra.run.smithi111.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-10-11T10:37:23.366 INFO:teuthology.orchestra.run.smithi050.stderr:bash: ntpq: command not found 2023-10-11T10:37:23.370 INFO:teuthology.orchestra.run.smithi050.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-10-11T10:37:23.371 INFO:teuthology.orchestra.run.smithi050.stdout:=============================================================================== 2023-10-11T10:37:23.371 INFO:teuthology.orchestra.run.smithi050.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-11T10:37:23.371 INFO:teuthology.orchestra.run.smithi050.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-11T10:37:23.371 INFO:teuthology.orchestra.run.smithi050.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-11T10:37:23.371 INFO:teuthology.orchestra.run.smithi050.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-11T10:37:23.402 INFO:teuthology.orchestra.run.smithi111.stderr:bash: ntpq: command not found 2023-10-11T10:37:23.405 INFO:teuthology.orchestra.run.smithi111.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-10-11T10:37:23.406 INFO:teuthology.orchestra.run.smithi111.stdout:=============================================================================== 2023-10-11T10:37:23.406 INFO:teuthology.orchestra.run.smithi111.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-11T10:37:23.406 INFO:teuthology.orchestra.run.smithi111.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-11T10:37:23.406 INFO:teuthology.orchestra.run.smithi111.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-11T10:37:23.407 INFO:teuthology.orchestra.run.smithi111.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-11T10:37:23.408 INFO:teuthology.run_tasks:Running task install... 2023-10-11T10:37:23.416 DEBUG:teuthology.task.install:project ceph 2023-10-11T10:37:23.417 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '4bf94f59187ed7eba271559c0898cec750ad93d5'}} 2023-10-11T10:37:23.417 DEBUG:teuthology.task.install:config {'extra_packages': ['rbd-mirror'], 'flavor': 'default', 'sha1': '4bf94f59187ed7eba271559c0898cec750ad93d5'} 2023-10-11T10:37:23.418 INFO:teuthology.task.install:Using flavor: default 2023-10-11T10:37:23.423 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-10-11T10:37:23.424 INFO:teuthology.task.install:extra packages: ['rbd-mirror'] 2023-10-11T10:37:23.424 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': ['rbd-mirror'], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '4bf94f59187ed7eba271559c0898cec750ad93d5', 'tag': None, 'wait_for_package': False} 2023-10-11T10:37:23.425 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=4bf94f59187ed7eba271559c0898cec750ad93d5 2023-10-11T10:37:23.427 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': ['rbd-mirror'], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '4bf94f59187ed7eba271559c0898cec750ad93d5', 'tag': None, 'wait_for_package': False} 2023-10-11T10:37:23.427 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=4bf94f59187ed7eba271559c0898cec750ad93d5 2023-10-11T10:37:23.674 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-dis-testing/4bf94f59187ed7eba271559c0898cec750ad93d5/centos/8/flavors/default/ 2023-10-11T10:37:23.675 INFO:teuthology.task.install.rpm:Package version is 18.0.0-6661.g4bf94f59 2023-10-11T10:37:23.676 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-dis-testing/4bf94f59187ed7eba271559c0898cec750ad93d5/centos/8/flavors/default/ 2023-10-11T10:37:23.676 INFO:teuthology.task.install.rpm:Package version is 18.0.0-6661.g4bf94f59 2023-10-11T10:37:23.785 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-dis-testing/4bf94f59187ed7eba271559c0898cec750ad93d5/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-dis-testing/4bf94f59187ed7eba271559c0898cec750ad93d5/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-dis-testing/4bf94f59187ed7eba271559c0898cec750ad93d5/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-10-11T10:37:23.786 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2023-10-11T10:37:23.786 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-10-11T10:37:23.817 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, rbd-mirror on remote rpm x86_64 2023-10-11T10:37:23.817 DEBUG:teuthology.orchestra.run.smithi050:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/4bf94f59187ed7eba271559c0898cec750ad93d5/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-10-11T10:37:23.871 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-dis-testing/4bf94f59187ed7eba271559c0898cec750ad93d5/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-dis-testing/4bf94f59187ed7eba271559c0898cec750ad93d5/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-dis-testing/4bf94f59187ed7eba271559c0898cec750ad93d5/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-10-11T10:37:23.871 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2023-10-11T10:37:23.872 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-10-11T10:37:23.889 DEBUG:teuthology.orchestra.run.smithi050:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-10-11T10:37:23.899 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, rbd-mirror on remote rpm x86_64 2023-10-11T10:37:23.899 DEBUG:teuthology.orchestra.run.smithi111:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/4bf94f59187ed7eba271559c0898cec750ad93d5/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-10-11T10:37:23.966 DEBUG:teuthology.orchestra.run.smithi111:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-10-11T10:37:23.969 DEBUG:teuthology.orchestra.run.smithi050:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-10-11T10:37:24.036 INFO:teuthology.orchestra.run.smithi050.stdout:check_obsoletes = 1 2023-10-11T10:37:24.038 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum clean all 2023-10-11T10:37:24.043 DEBUG:teuthology.orchestra.run.smithi111:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-10-11T10:37:24.109 INFO:teuthology.orchestra.run.smithi111.stdout:check_obsoletes = 1 2023-10-11T10:37:24.110 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum clean all 2023-10-11T10:37:24.432 INFO:teuthology.orchestra.run.smithi050.stdout:Updating Subscription Management repositories. 2023-10-11T10:37:24.479 INFO:teuthology.orchestra.run.smithi111.stdout:Updating Subscription Management repositories. 2023-10-11T10:37:24.701 INFO:teuthology.orchestra.run.smithi050.stdout:44 files removed 2023-10-11T10:37:24.715 INFO:teuthology.orchestra.run.smithi111.stdout:44 files removed 2023-10-11T10:37:24.749 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install ceph-radosgw 2023-10-11T10:37:24.761 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install ceph-radosgw 2023-10-11T10:37:25.089 INFO:teuthology.orchestra.run.smithi111.stdout:Updating Subscription Management repositories. 2023-10-11T10:37:25.098 INFO:teuthology.orchestra.run.smithi050.stdout:Updating Subscription Management repositories. 2023-10-11T10:37:25.771 INFO:teuthology.orchestra.run.smithi111.stdout:ceph packages for x86_64 240 kB/s | 81 kB 00:00 2023-10-11T10:37:25.913 INFO:teuthology.orchestra.run.smithi050.stdout:ceph packages for x86_64 180 kB/s | 81 kB 00:00 2023-10-11T10:37:26.072 INFO:teuthology.orchestra.run.smithi111.stdout:ceph noarch packages 35 kB/s | 9.4 kB 00:00 2023-10-11T10:37:26.239 INFO:teuthology.orchestra.run.smithi050.stdout:ceph noarch packages 32 kB/s | 9.4 kB 00:00 2023-10-11T10:37:26.415 INFO:teuthology.orchestra.run.smithi111.stdout:ceph source packages 4.7 kB/s | 1.5 kB 00:00 2023-10-11T10:37:26.565 INFO:teuthology.orchestra.run.smithi111.stdout:Copr repo for python3-asyncssh owned by ceph 26 kB/s | 3.5 kB 00:00 2023-10-11T10:37:26.590 INFO:teuthology.orchestra.run.smithi050.stdout:ceph source packages 4.6 kB/s | 1.5 kB 00:00 2023-10-11T10:37:26.749 INFO:teuthology.orchestra.run.smithi050.stdout:Copr repo for python3-asyncssh owned by ceph 25 kB/s | 3.5 kB 00:00 2023-10-11T10:37:27.499 INFO:teuthology.orchestra.run.smithi111.stdout:Extra Packages for Enterprise Linux 17 MB/s | 16 MB 00:00 2023-10-11T10:37:28.909 INFO:teuthology.orchestra.run.smithi050.stdout:Extra Packages for Enterprise Linux 7.4 MB/s | 16 MB 00:02 2023-10-11T10:37:31.025 INFO:teuthology.orchestra.run.smithi111.stdout:lab-extras 500 kB/s | 24 kB 00:00 2023-10-11T10:37:31.976 INFO:teuthology.orchestra.run.smithi111.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 60 MB/s | 53 MB 00:00 2023-10-11T10:37:32.658 INFO:teuthology.orchestra.run.smithi050.stdout:lab-extras 549 kB/s | 24 kB 00:00 2023-10-11T10:37:33.584 INFO:teuthology.orchestra.run.smithi050.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 62 MB/s | 53 MB 00:00 2023-10-11T10:37:40.186 INFO:teuthology.orchestra.run.smithi111.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 53 MB/s | 47 MB 00:00 2023-10-11T10:37:42.185 INFO:teuthology.orchestra.run.smithi050.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 56 MB/s | 47 MB 00:00 2023-10-11T10:37:51.370 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2023-10-11T10:37:51.372 INFO:teuthology.orchestra.run.smithi111.stdout:==================================================================================================== 2023-10-11T10:37:51.372 INFO:teuthology.orchestra.run.smithi111.stdout: Package Arch Version Repository Size 2023-10-11T10:37:51.373 INFO:teuthology.orchestra.run.smithi111.stdout:==================================================================================================== 2023-10-11T10:37:51.373 INFO:teuthology.orchestra.run.smithi111.stdout:Installing: 2023-10-11T10:37:51.373 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-radosgw x86_64 2:18.0.0-6661.g4bf94f59.el8 ceph 14 M 2023-10-11T10:37:51.373 INFO:teuthology.orchestra.run.smithi111.stdout:Upgrading: 2023-10-11T10:37:51.373 INFO:teuthology.orchestra.run.smithi111.stdout: librados2 x86_64 2:18.0.0-6661.g4bf94f59.el8 ceph 3.7 M 2023-10-11T10:37:51.374 INFO:teuthology.orchestra.run.smithi111.stdout: librbd1 x86_64 2:18.0.0-6661.g4bf94f59.el8 ceph 3.7 M 2023-10-11T10:37:51.374 INFO:teuthology.orchestra.run.smithi111.stdout:Installing dependencies: 2023-10-11T10:37:51.374 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-base x86_64 2:18.0.0-6661.g4bf94f59.el8 ceph 5.4 M 2023-10-11T10:37:51.374 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-common x86_64 2:18.0.0-6661.g4bf94f59.el8 ceph 23 M 2023-10-11T10:37:51.374 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-selinux x86_64 2:18.0.0-6661.g4bf94f59.el8 ceph 24 k 2023-10-11T10:37:51.375 INFO:teuthology.orchestra.run.smithi111.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-10-11T10:37:51.375 INFO:teuthology.orchestra.run.smithi111.stdout: libcephfs2 x86_64 2:18.0.0-6661.g4bf94f59.el8 ceph 840 k 2023-10-11T10:37:51.375 INFO:teuthology.orchestra.run.smithi111.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-10-11T10:37:51.375 INFO:teuthology.orchestra.run.smithi111.stdout: librabbitmq x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2023-10-11T10:37:51.375 INFO:teuthology.orchestra.run.smithi111.stdout: libradosstriper1 x86_64 2:18.0.0-6661.g4bf94f59.el8 ceph 518 k 2023-10-11T10:37:51.376 INFO:teuthology.orchestra.run.smithi111.stdout: librdkafka x86_64 0.11.4-3.el8 rhel-8-for-x86_64-appstream-rpms 354 k 2023-10-11T10:37:51.376 INFO:teuthology.orchestra.run.smithi111.stdout: librgw2 x86_64 2:18.0.0-6661.g4bf94f59.el8 ceph 6.9 M 2023-10-11T10:37:51.376 INFO:teuthology.orchestra.run.smithi111.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-10-11T10:37:51.376 INFO:teuthology.orchestra.run.smithi111.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2023-10-11T10:37:51.376 INFO:teuthology.orchestra.run.smithi111.stdout: python3-ceph-argparse x86_64 2:18.0.0-6661.g4bf94f59.el8 ceph 45 k 2023-10-11T10:37:51.376 INFO:teuthology.orchestra.run.smithi111.stdout: python3-ceph-common x86_64 2:18.0.0-6661.g4bf94f59.el8 ceph 125 k 2023-10-11T10:37:51.377 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cephfs x86_64 2:18.0.0-6661.g4bf94f59.el8 ceph 197 k 2023-10-11T10:37:51.377 INFO:teuthology.orchestra.run.smithi111.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2023-10-11T10:37:51.377 INFO:teuthology.orchestra.run.smithi111.stdout: python3-rados x86_64 2:18.0.0-6661.g4bf94f59.el8 ceph 370 k 2023-10-11T10:37:51.377 INFO:teuthology.orchestra.run.smithi111.stdout: python3-rbd x86_64 2:18.0.0-6661.g4bf94f59.el8 ceph 363 k 2023-10-11T10:37:51.377 INFO:teuthology.orchestra.run.smithi111.stdout: python3-rgw x86_64 2:18.0.0-6661.g4bf94f59.el8 ceph 107 k 2023-10-11T10:37:51.378 INFO:teuthology.orchestra.run.smithi111.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-10-11T10:37:51.378 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T10:37:51.378 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2023-10-11T10:37:51.378 INFO:teuthology.orchestra.run.smithi111.stdout:==================================================================================================== 2023-10-11T10:37:51.378 INFO:teuthology.orchestra.run.smithi111.stdout:Install 21 Packages 2023-10-11T10:37:51.379 INFO:teuthology.orchestra.run.smithi111.stdout:Upgrade 2 Packages 2023-10-11T10:37:51.379 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T10:37:51.379 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 62 M 2023-10-11T10:37:51.380 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2023-10-11T10:37:52.778 INFO:teuthology.orchestra.run.smithi111.stdout:(1/23): ceph-common-18.0.0-6661.g4bf94f59.el8.x 17 MB/s | 23 MB 00:01 2023-10-11T10:37:52.820 INFO:teuthology.orchestra.run.smithi111.stdout:(2/23): ceph-selinux-18.0.0-6661.g4bf94f59.el8. 584 kB/s | 24 kB 00:00 2023-10-11T10:37:52.905 INFO:teuthology.orchestra.run.smithi111.stdout:(3/23): libcephfs2-18.0.0-6661.g4bf94f59.el8.x8 9.9 MB/s | 840 kB 00:00 2023-10-11T10:37:52.962 INFO:teuthology.orchestra.run.smithi111.stdout:(4/23): libradosstriper1-18.0.0-6661.g4bf94f59. 8.7 MB/s | 518 kB 00:00 2023-10-11T10:37:53.062 INFO:teuthology.orchestra.run.smithi111.stdout:(5/23): ceph-base-18.0.0-6661.g4bf94f59.el8.x86 3.3 MB/s | 5.4 MB 00:01 2023-10-11T10:37:53.145 INFO:teuthology.orchestra.run.smithi111.stdout:(6/23): python3-ceph-argparse-18.0.0-6661.g4bf9 538 kB/s | 45 kB 00:00 2023-10-11T10:37:53.246 INFO:teuthology.orchestra.run.smithi111.stdout:(7/23): python3-ceph-common-18.0.0-6661.g4bf94f 1.2 MB/s | 125 kB 00:00 2023-10-11T10:37:53.329 INFO:teuthology.orchestra.run.smithi111.stdout:(8/23): librgw2-18.0.0-6661.g4bf94f59.el8.x86_6 19 MB/s | 6.9 MB 00:00 2023-10-11T10:37:53.388 INFO:teuthology.orchestra.run.smithi111.stdout:(9/23): python3-cephfs-18.0.0-6661.g4bf94f59.el 1.4 MB/s | 197 kB 00:00 2023-10-11T10:37:53.414 INFO:teuthology.orchestra.run.smithi111.stdout:(10/23): python3-rados-18.0.0-6661.g4bf94f59.el 4.3 MB/s | 370 kB 00:00 2023-10-11T10:37:53.456 INFO:teuthology.orchestra.run.smithi111.stdout:(11/23): python3-rbd-18.0.0-6661.g4bf94f59.el8. 5.3 MB/s | 363 kB 00:00 2023-10-11T10:37:53.472 INFO:teuthology.orchestra.run.smithi111.stdout:(12/23): python3-rgw-18.0.0-6661.g4bf94f59.el8. 1.8 MB/s | 107 kB 00:00 2023-10-11T10:37:53.831 INFO:teuthology.orchestra.run.smithi111.stdout:(13/23): liboath-2.6.2-3.el8.x86_64.rpm 164 kB/s | 59 kB 00:00 2023-10-11T10:37:53.846 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2023-10-11T10:37:53.848 INFO:teuthology.orchestra.run.smithi050.stdout:==================================================================================================== 2023-10-11T10:37:53.849 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repository Size 2023-10-11T10:37:53.849 INFO:teuthology.orchestra.run.smithi050.stdout:==================================================================================================== 2023-10-11T10:37:53.849 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2023-10-11T10:37:53.849 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-radosgw x86_64 2:18.0.0-6661.g4bf94f59.el8 ceph 14 M 2023-10-11T10:37:53.850 INFO:teuthology.orchestra.run.smithi050.stdout:Upgrading: 2023-10-11T10:37:53.850 INFO:teuthology.orchestra.run.smithi050.stdout: librados2 x86_64 2:18.0.0-6661.g4bf94f59.el8 ceph 3.7 M 2023-10-11T10:37:53.850 INFO:teuthology.orchestra.run.smithi050.stdout: librbd1 x86_64 2:18.0.0-6661.g4bf94f59.el8 ceph 3.7 M 2023-10-11T10:37:53.850 INFO:teuthology.orchestra.run.smithi050.stdout:Installing dependencies: 2023-10-11T10:37:53.850 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-base x86_64 2:18.0.0-6661.g4bf94f59.el8 ceph 5.4 M 2023-10-11T10:37:53.851 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-common x86_64 2:18.0.0-6661.g4bf94f59.el8 ceph 23 M 2023-10-11T10:37:53.851 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-selinux x86_64 2:18.0.0-6661.g4bf94f59.el8 ceph 24 k 2023-10-11T10:37:53.851 INFO:teuthology.orchestra.run.smithi050.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-10-11T10:37:53.851 INFO:teuthology.orchestra.run.smithi050.stdout: libcephfs2 x86_64 2:18.0.0-6661.g4bf94f59.el8 ceph 840 k 2023-10-11T10:37:53.851 INFO:teuthology.orchestra.run.smithi050.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-10-11T10:37:53.851 INFO:teuthology.orchestra.run.smithi050.stdout: librabbitmq x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2023-10-11T10:37:53.852 INFO:teuthology.orchestra.run.smithi050.stdout: libradosstriper1 x86_64 2:18.0.0-6661.g4bf94f59.el8 ceph 518 k 2023-10-11T10:37:53.852 INFO:teuthology.orchestra.run.smithi050.stdout: librdkafka x86_64 0.11.4-3.el8 rhel-8-for-x86_64-appstream-rpms 354 k 2023-10-11T10:37:53.852 INFO:teuthology.orchestra.run.smithi050.stdout: librgw2 x86_64 2:18.0.0-6661.g4bf94f59.el8 ceph 6.9 M 2023-10-11T10:37:53.852 INFO:teuthology.orchestra.run.smithi050.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-10-11T10:37:53.852 INFO:teuthology.orchestra.run.smithi050.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2023-10-11T10:37:53.853 INFO:teuthology.orchestra.run.smithi050.stdout: python3-ceph-argparse x86_64 2:18.0.0-6661.g4bf94f59.el8 ceph 45 k 2023-10-11T10:37:53.853 INFO:teuthology.orchestra.run.smithi050.stdout: python3-ceph-common x86_64 2:18.0.0-6661.g4bf94f59.el8 ceph 125 k 2023-10-11T10:37:53.853 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cephfs x86_64 2:18.0.0-6661.g4bf94f59.el8 ceph 197 k 2023-10-11T10:37:53.853 INFO:teuthology.orchestra.run.smithi050.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2023-10-11T10:37:53.853 INFO:teuthology.orchestra.run.smithi050.stdout: python3-rados x86_64 2:18.0.0-6661.g4bf94f59.el8 ceph 370 k 2023-10-11T10:37:53.854 INFO:teuthology.orchestra.run.smithi050.stdout: python3-rbd x86_64 2:18.0.0-6661.g4bf94f59.el8 ceph 363 k 2023-10-11T10:37:53.854 INFO:teuthology.orchestra.run.smithi050.stdout: python3-rgw x86_64 2:18.0.0-6661.g4bf94f59.el8 ceph 107 k 2023-10-11T10:37:53.854 INFO:teuthology.orchestra.run.smithi050.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-10-11T10:37:53.854 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-11T10:37:53.854 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2023-10-11T10:37:53.855 INFO:teuthology.orchestra.run.smithi050.stdout:==================================================================================================== 2023-10-11T10:37:53.855 INFO:teuthology.orchestra.run.smithi050.stdout:Install 21 Packages 2023-10-11T10:37:53.855 INFO:teuthology.orchestra.run.smithi050.stdout:Upgrade 2 Packages 2023-10-11T10:37:53.855 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-11T10:37:53.856 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 62 M 2023-10-11T10:37:53.856 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2023-10-11T10:37:53.982 INFO:teuthology.orchestra.run.smithi111.stdout:(14/23): gperftools-libs-2.7-9.el8.x86_64.rpm 583 kB/s | 306 kB 00:00 2023-10-11T10:37:54.024 INFO:teuthology.orchestra.run.smithi111.stdout:(15/23): libunwind-1.3.1-3.el8.x86_64.rpm 391 kB/s | 75 kB 00:00 2023-10-11T10:37:54.074 INFO:teuthology.orchestra.run.smithi111.stdout:(16/23): librabbitmq-0.9.0-3.el8.x86_64.rpm 937 kB/s | 47 kB 00:00 2023-10-11T10:37:54.116 INFO:teuthology.orchestra.run.smithi111.stdout:(17/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 6.1 MB/s | 259 kB 00:00 2023-10-11T10:37:54.149 INFO:teuthology.orchestra.run.smithi111.stdout:(18/23): python3-prettytable-0.7.2-14.el8.noarc 1.3 MB/s | 44 kB 00:00 2023-10-11T10:37:54.191 INFO:teuthology.orchestra.run.smithi111.stdout:(19/23): librdkafka-0.11.4-3.el8.x86_64.rpm 8.3 MB/s | 354 kB 00:00 2023-10-11T10:37:54.675 INFO:teuthology.orchestra.run.smithi111.stdout:(20/23): thrift-0.13.0-2.el8.x86_64.rpm 2.5 MB/s | 1.7 MB 00:00 2023-10-11T10:37:54.875 INFO:teuthology.orchestra.run.smithi111.stdout:(21/23): librados2-18.0.0-6661.g4bf94f59.el8.x8 5.5 MB/s | 3.7 MB 00:00 2023-10-11T10:37:55.085 INFO:teuthology.orchestra.run.smithi050.stdout:(1/23): ceph-base-18.0.0-6661.g4bf94f59.el8.x86 4.5 MB/s | 5.4 MB 00:01 2023-10-11T10:37:55.144 INFO:teuthology.orchestra.run.smithi050.stdout:(2/23): ceph-selinux-18.0.0-6661.g4bf94f59.el8. 418 kB/s | 24 kB 00:00 2023-10-11T10:37:55.201 INFO:teuthology.orchestra.run.smithi111.stdout:(22/23): librbd1-18.0.0-6661.g4bf94f59.el8.x86_ 7.1 MB/s | 3.7 MB 00:00 2023-10-11T10:37:55.294 INFO:teuthology.orchestra.run.smithi050.stdout:(3/23): libcephfs2-18.0.0-6661.g4bf94f59.el8.x8 5.5 MB/s | 840 kB 00:00 2023-10-11T10:37:55.411 INFO:teuthology.orchestra.run.smithi050.stdout:(4/23): libradosstriper1-18.0.0-6661.g4bf94f59. 4.3 MB/s | 518 kB 00:00 2023-10-11T10:37:55.417 INFO:teuthology.orchestra.run.smithi111.stdout:(23/23): ceph-radosgw-18.0.0-6661.g4bf94f59.el8 3.5 MB/s | 14 MB 00:04 2023-10-11T10:37:55.420 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2023-10-11T10:37:55.420 INFO:teuthology.orchestra.run.smithi111.stdout:Total 15 MB/s | 62 MB 00:04 2023-10-11T10:37:55.468 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2023-10-11T10:37:55.516 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2023-10-11T10:37:55.516 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2023-10-11T10:37:55.864 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2023-10-11T10:37:55.867 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2023-10-11T10:37:55.945 INFO:teuthology.orchestra.run.smithi050.stdout:(5/23): ceph-radosgw-18.0.0-6661.g4bf94f59.el8. 6.8 MB/s | 14 MB 00:02 2023-10-11T10:37:56.020 INFO:teuthology.orchestra.run.smithi050.stdout:(6/23): python3-ceph-argparse-18.0.0-6661.g4bf9 597 kB/s | 45 kB 00:00 2023-10-11T10:37:56.096 INFO:teuthology.orchestra.run.smithi050.stdout:(7/23): python3-ceph-common-18.0.0-6661.g4bf94f 1.6 MB/s | 125 kB 00:00 2023-10-11T10:37:56.179 INFO:teuthology.orchestra.run.smithi050.stdout:(8/23): librgw2-18.0.0-6661.g4bf94f59.el8.x86_6 9.0 MB/s | 6.9 MB 00:00 2023-10-11T10:37:56.204 INFO:teuthology.orchestra.run.smithi050.stdout:(9/23): python3-cephfs-18.0.0-6661.g4bf94f59.el 1.8 MB/s | 197 kB 00:00 2023-10-11T10:37:56.246 INFO:teuthology.orchestra.run.smithi050.stdout:(10/23): python3-rados-18.0.0-6661.g4bf94f59.el 5.4 MB/s | 370 kB 00:00 2023-10-11T10:37:56.272 INFO:teuthology.orchestra.run.smithi050.stdout:(11/23): python3-rbd-18.0.0-6661.g4bf94f59.el8. 5.3 MB/s | 363 kB 00:00 2023-10-11T10:37:56.305 INFO:teuthology.orchestra.run.smithi050.stdout:(12/23): python3-rgw-18.0.0-6661.g4bf94f59.el8. 1.8 MB/s | 107 kB 00:00 2023-10-11T10:37:56.706 INFO:teuthology.orchestra.run.smithi050.stdout:(13/23): ceph-common-18.0.0-6661.g4bf94f59.el8. 8.0 MB/s | 23 MB 00:02 2023-10-11T10:37:56.723 INFO:teuthology.orchestra.run.smithi050.stdout:(14/23): liboath-2.6.2-3.el8.x86_64.rpm 141 kB/s | 59 kB 00:00 2023-10-11T10:37:56.806 INFO:teuthology.orchestra.run.smithi050.stdout:(15/23): gperftools-libs-2.7-9.el8.x86_64.rpm 573 kB/s | 306 kB 00:00 2023-10-11T10:37:56.856 INFO:teuthology.orchestra.run.smithi050.stdout:(16/23): librabbitmq-0.9.0-3.el8.x86_64.rpm 953 kB/s | 47 kB 00:00 2023-10-11T10:37:56.898 INFO:teuthology.orchestra.run.smithi050.stdout:(17/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 6.1 MB/s | 259 kB 00:00 2023-10-11T10:37:56.924 INFO:teuthology.orchestra.run.smithi050.stdout:(18/23): python3-prettytable-0.7.2-14.el8.noarc 1.7 MB/s | 44 kB 00:00 2023-10-11T10:37:56.949 INFO:teuthology.orchestra.run.smithi050.stdout:(19/23): libunwind-1.3.1-3.el8.x86_64.rpm 309 kB/s | 75 kB 00:00 2023-10-11T10:37:56.983 INFO:teuthology.orchestra.run.smithi050.stdout:(20/23): librdkafka-0.11.4-3.el8.x86_64.rpm 5.9 MB/s | 354 kB 00:00 2023-10-11T10:37:57.025 INFO:teuthology.orchestra.run.smithi050.stdout:(21/23): thrift-0.13.0-2.el8.x86_64.rpm 5.8 MB/s | 1.7 MB 00:00 2023-10-11T10:37:57.124 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2023-10-11T10:37:57.308 INFO:teuthology.orchestra.run.smithi050.stdout:(22/23): librados2-18.0.0-6661.g4bf94f59.el8.x8 10 MB/s | 3.7 MB 00:00 2023-10-11T10:37:57.366 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : thrift-0.13.0-2.el8.x86_64 1/25 2023-10-11T10:37:57.375 INFO:teuthology.orchestra.run.smithi050.stdout:(23/23): librbd1-18.0.0-6661.g4bf94f59.el8.x86_ 9.5 MB/s | 3.7 MB 00:00 2023-10-11T10:37:57.379 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2023-10-11T10:37:57.379 INFO:teuthology.orchestra.run.smithi050.stdout:Total 17 MB/s | 62 MB 00:03 2023-10-11T10:37:57.422 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2023-10-11T10:37:57.470 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2023-10-11T10:37:57.471 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2023-10-11T10:37:57.555 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 2/25 2023-10-11T10:37:57.830 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2023-10-11T10:37:57.834 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2023-10-11T10:37:57.950 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 2/25 2023-10-11T10:37:58.185 INFO:teuthology.orchestra.run.smithi111.stdout: Upgrading : librados2-2:18.0.0-6661.g4bf94f59.el8.x86_64 3/25 2023-10-11T10:37:58.285 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: librados2-2:18.0.0-6661.g4bf94f59.el8.x86_64 3/25 2023-10-11T10:37:58.491 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-rados-2:18.0.0-6661.g4bf94f59.el8.x86_64 4/25 2023-10-11T10:37:58.721 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : liboath-2.6.2-3.el8.x86_64 5/25 2023-10-11T10:37:58.927 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : libcephfs2-2:18.0.0-6661.g4bf94f59.el8.x86_64 6/25 2023-10-11T10:37:59.035 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2023-10-11T10:37:59.277 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : thrift-0.13.0-2.el8.x86_64 1/25 2023-10-11T10:37:59.338 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: libcephfs2-2:18.0.0-6661.g4bf94f59.el8.x86_64 6/25 2023-10-11T10:37:59.462 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 2/25 2023-10-11T10:37:59.677 INFO:teuthology.orchestra.run.smithi111.stdout: Upgrading : librbd1-2:18.0.0-6661.g4bf94f59.el8.x86_64 7/25 2023-10-11T10:37:59.751 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: librbd1-2:18.0.0-6661.g4bf94f59.el8.x86_64 7/25 2023-10-11T10:37:59.883 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 2/25 2023-10-11T10:37:59.893 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 8/25 2023-10-11T10:37:59.992 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 8/25 2023-10-11T10:38:00.070 INFO:teuthology.orchestra.run.smithi050.stdout: Upgrading : librados2-2:18.0.0-6661.g4bf94f59.el8.x86_64 3/25 2023-10-11T10:38:00.142 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 9/25 2023-10-11T10:38:00.168 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: librados2-2:18.0.0-6661.g4bf94f59.el8.x86_64 3/25 2023-10-11T10:38:00.323 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-ceph-argparse-2:18.0.0-6661.g4bf94f59.el8. 10/25 2023-10-11T10:38:00.354 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-rados-2:18.0.0-6661.g4bf94f59.el8.x86_64 4/25 2023-10-11T10:38:00.505 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-cephfs-2:18.0.0-6661.g4bf94f59.el8.x86_64 11/25 2023-10-11T10:38:00.757 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-rbd-2:18.0.0-6661.g4bf94f59.el8.x86_64 12/25 2023-10-11T10:38:00.918 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : libradosstriper1-2:18.0.0-6661.g4bf94f59.el8.x86_6 13/25 2023-10-11T10:38:00.975 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: libradosstriper1-2:18.0.0-6661.g4bf94f59.el8.x86_6 13/25 2023-10-11T10:38:01.096 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : liboath-2.6.2-3.el8.x86_64 5/25 2023-10-11T10:38:01.112 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 14/25 2023-10-11T10:38:01.270 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : libcephfs2-2:18.0.0-6661.g4bf94f59.el8.x86_64 6/25 2023-10-11T10:38:01.347 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 15/25 2023-10-11T10:38:01.688 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: libcephfs2-2:18.0.0-6661.g4bf94f59.el8.x86_64 6/25 2023-10-11T10:38:01.853 INFO:teuthology.orchestra.run.smithi050.stdout: Upgrading : librbd1-2:18.0.0-6661.g4bf94f59.el8.x86_64 7/25 2023-10-11T10:38:01.936 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: librbd1-2:18.0.0-6661.g4bf94f59.el8.x86_64 7/25 2023-10-11T10:38:02.086 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 8/25 2023-10-11T10:38:02.158 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 8/25 2023-10-11T10:38:02.168 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 16/25 2023-10-11T10:38:02.326 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : librgw2-2:18.0.0-6661.g4bf94f59.el8.x86_64 17/25 2023-10-11T10:38:02.343 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 9/25 2023-10-11T10:38:02.431 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: librgw2-2:18.0.0-6661.g4bf94f59.el8.x86_64 17/25 2023-10-11T10:38:02.501 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-ceph-argparse-2:18.0.0-6661.g4bf94f59.el8. 10/25 2023-10-11T10:38:02.635 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-rgw-2:18.0.0-6661.g4bf94f59.el8.x86_64 18/25 2023-10-11T10:38:02.733 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-cephfs-2:18.0.0-6661.g4bf94f59.el8.x86_64 11/25 2023-10-11T10:38:02.860 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-ceph-common-2:18.0.0-6661.g4bf94f59.el8.x8 19/25 2023-10-11T10:38:02.961 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-rbd-2:18.0.0-6661.g4bf94f59.el8.x86_64 12/25 2023-10-11T10:38:03.145 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : libradosstriper1-2:18.0.0-6661.g4bf94f59.el8.x86_6 13/25 2023-10-11T10:38:03.216 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: libradosstriper1-2:18.0.0-6661.g4bf94f59.el8.x86_6 13/25 2023-10-11T10:38:03.347 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 14/25 2023-10-11T10:38:03.546 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 15/25 2023-10-11T10:38:04.432 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 16/25 2023-10-11T10:38:04.653 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : librgw2-2:18.0.0-6661.g4bf94f59.el8.x86_64 17/25 2023-10-11T10:38:04.714 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: librgw2-2:18.0.0-6661.g4bf94f59.el8.x86_64 17/25 2023-10-11T10:38:04.932 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-rgw-2:18.0.0-6661.g4bf94f59.el8.x86_64 18/25 2023-10-11T10:38:05.103 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-ceph-common-2:18.0.0-6661.g4bf94f59.el8.x8 19/25 2023-10-11T10:38:06.622 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-common-2:18.0.0-6661.g4bf94f59.el8.x86_64 20/25 2023-10-11T10:38:06.842 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-common-2:18.0.0-6661.g4bf94f59.el8.x86_64 20/25 2023-10-11T10:38:07.465 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-common-2:18.0.0-6661.g4bf94f59.el8.x86_64 20/25 2023-10-11T10:38:07.753 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-base-2:18.0.0-6661.g4bf94f59.el8.x86_64 21/25 2023-10-11T10:38:07.859 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-base-2:18.0.0-6661.g4bf94f59.el8.x86_64 21/25 2023-10-11T10:38:08.033 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-selinux-2:18.0.0-6661.g4bf94f59.el8.x86_64 22/25 2023-10-11T10:38:09.089 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-common-2:18.0.0-6661.g4bf94f59.el8.x86_64 20/25 2023-10-11T10:38:09.352 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-common-2:18.0.0-6661.g4bf94f59.el8.x86_64 20/25 2023-10-11T10:38:09.991 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-common-2:18.0.0-6661.g4bf94f59.el8.x86_64 20/25 2023-10-11T10:38:10.261 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-base-2:18.0.0-6661.g4bf94f59.el8.x86_64 21/25 2023-10-11T10:38:10.358 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-base-2:18.0.0-6661.g4bf94f59.el8.x86_64 21/25 2023-10-11T10:38:10.527 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-selinux-2:18.0.0-6661.g4bf94f59.el8.x86_64 22/25 2023-10-11T10:38:26.856 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-selinux-2:18.0.0-6661.g4bf94f59.el8.x86_64 22/25 2023-10-11T10:38:26.856 INFO:teuthology.orchestra.run.smithi111.stdout:skipping the directory /sys 2023-10-11T10:38:26.857 INFO:teuthology.orchestra.run.smithi111.stdout:skipping the directory /proc 2023-10-11T10:38:26.857 INFO:teuthology.orchestra.run.smithi111.stdout:skipping the directory /dev 2023-10-11T10:38:26.857 INFO:teuthology.orchestra.run.smithi111.stdout:skipping the directory /run 2023-10-11T10:38:26.857 INFO:teuthology.orchestra.run.smithi111.stdout:skipping the directory /mnt 2023-10-11T10:38:26.857 INFO:teuthology.orchestra.run.smithi111.stdout:skipping the directory /var/tmp 2023-10-11T10:38:26.858 INFO:teuthology.orchestra.run.smithi111.stdout:skipping the directory /home 2023-10-11T10:38:26.858 INFO:teuthology.orchestra.run.smithi111.stdout:skipping the directory /tmp 2023-10-11T10:38:26.858 INFO:teuthology.orchestra.run.smithi111.stdout:skipping the directory /dev 2023-10-11T10:38:26.858 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T10:38:28.284 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-radosgw-2:18.0.0-6661.g4bf94f59.el8.x86_64 23/25 2023-10-11T10:38:28.304 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-6661.g4bf94f59.el8.x86_64 23/25 2023-10-11T10:38:28.305 INFO:teuthology.orchestra.run.smithi111.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2023-10-11T10:38:28.525 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2023-10-11T10:38:28.525 INFO:teuthology.orchestra.run.smithi111.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2023-10-11T10:38:29.162 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2023-10-11T10:38:29.162 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-base-2:18.0.0-6661.g4bf94f59.el8.x86_64 1/25 2023-10-11T10:38:29.162 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-common-2:18.0.0-6661.g4bf94f59.el8.x86_64 2/25 2023-10-11T10:38:29.166 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-radosgw-2:18.0.0-6661.g4bf94f59.el8.x86_64 3/25 2023-10-11T10:38:29.166 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-selinux-2:18.0.0-6661.g4bf94f59.el8.x86_64 4/25 2023-10-11T10:38:29.167 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : libcephfs2-2:18.0.0-6661.g4bf94f59.el8.x86_64 5/25 2023-10-11T10:38:29.167 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : libradosstriper1-2:18.0.0-6661.g4bf94f59.el8.x86_6 6/25 2023-10-11T10:38:29.167 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : librgw2-2:18.0.0-6661.g4bf94f59.el8.x86_64 7/25 2023-10-11T10:38:29.167 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-ceph-argparse-2:18.0.0-6661.g4bf94f59.el8. 8/25 2023-10-11T10:38:29.167 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-ceph-common-2:18.0.0-6661.g4bf94f59.el8.x8 9/25 2023-10-11T10:38:29.168 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-cephfs-2:18.0.0-6661.g4bf94f59.el8.x86_64 10/25 2023-10-11T10:38:29.168 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-rados-2:18.0.0-6661.g4bf94f59.el8.x86_64 11/25 2023-10-11T10:38:29.169 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-rbd-2:18.0.0-6661.g4bf94f59.el8.x86_64 12/25 2023-10-11T10:38:29.170 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-rgw-2:18.0.0-6661.g4bf94f59.el8.x86_64 13/25 2023-10-11T10:38:29.170 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 14/25 2023-10-11T10:38:29.170 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 15/25 2023-10-11T10:38:29.170 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 16/25 2023-10-11T10:38:29.170 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 17/25 2023-10-11T10:38:29.171 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 18/25 2023-10-11T10:38:29.171 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 19/25 2023-10-11T10:38:29.171 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 20/25 2023-10-11T10:38:29.171 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 21/25 2023-10-11T10:38:29.171 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : librados2-2:18.0.0-6661.g4bf94f59.el8.x86_64 22/25 2023-10-11T10:38:29.172 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2023-10-11T10:38:29.172 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : librbd1-2:18.0.0-6661.g4bf94f59.el8.x86_64 24/25 2023-10-11T10:38:30.503 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2023-10-11T10:38:30.503 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:00:51 ago on Wed 11 Oct 2023 10:37:39 AM UTC. 2023-10-11T10:38:30.720 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-selinux-2:18.0.0-6661.g4bf94f59.el8.x86_64 22/25 2023-10-11T10:38:30.720 INFO:teuthology.orchestra.run.smithi050.stdout:skipping the directory /sys 2023-10-11T10:38:30.720 INFO:teuthology.orchestra.run.smithi050.stdout:skipping the directory /proc 2023-10-11T10:38:30.720 INFO:teuthology.orchestra.run.smithi050.stdout:skipping the directory /dev 2023-10-11T10:38:30.721 INFO:teuthology.orchestra.run.smithi050.stdout:skipping the directory /run 2023-10-11T10:38:30.721 INFO:teuthology.orchestra.run.smithi050.stdout:skipping the directory /mnt 2023-10-11T10:38:30.721 INFO:teuthology.orchestra.run.smithi050.stdout:skipping the directory /var/tmp 2023-10-11T10:38:30.721 INFO:teuthology.orchestra.run.smithi050.stdout:skipping the directory /home 2023-10-11T10:38:30.721 INFO:teuthology.orchestra.run.smithi050.stdout:skipping the directory /tmp 2023-10-11T10:38:30.721 INFO:teuthology.orchestra.run.smithi050.stdout:skipping the directory /dev 2023-10-11T10:38:30.722 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-11T10:38:32.208 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-radosgw-2:18.0.0-6661.g4bf94f59.el8.x86_64 23/25 2023-10-11T10:38:32.229 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-6661.g4bf94f59.el8.x86_64 23/25 2023-10-11T10:38:32.229 INFO:teuthology.orchestra.run.smithi050.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2023-10-11T10:38:32.435 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2023-10-11T10:38:32.435 INFO:teuthology.orchestra.run.smithi050.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2023-10-11T10:38:32.839 INFO:teuthology.orchestra.run.smithi111.stdout:Installed products updated. 2023-10-11T10:38:33.041 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T10:38:33.042 INFO:teuthology.orchestra.run.smithi111.stdout:Upgraded: 2023-10-11T10:38:33.042 INFO:teuthology.orchestra.run.smithi111.stdout: librados2-2:18.0.0-6661.g4bf94f59.el8.x86_64 2023-10-11T10:38:33.042 INFO:teuthology.orchestra.run.smithi111.stdout: librbd1-2:18.0.0-6661.g4bf94f59.el8.x86_64 2023-10-11T10:38:33.042 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2023-10-11T10:38:33.043 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-base-2:18.0.0-6661.g4bf94f59.el8.x86_64 2023-10-11T10:38:33.043 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-common-2:18.0.0-6661.g4bf94f59.el8.x86_64 2023-10-11T10:38:33.043 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-radosgw-2:18.0.0-6661.g4bf94f59.el8.x86_64 2023-10-11T10:38:33.043 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-selinux-2:18.0.0-6661.g4bf94f59.el8.x86_64 2023-10-11T10:38:33.043 INFO:teuthology.orchestra.run.smithi111.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-10-11T10:38:33.044 INFO:teuthology.orchestra.run.smithi111.stdout: libcephfs2-2:18.0.0-6661.g4bf94f59.el8.x86_64 2023-10-11T10:38:33.044 INFO:teuthology.orchestra.run.smithi111.stdout: liboath-2.6.2-3.el8.x86_64 2023-10-11T10:38:33.044 INFO:teuthology.orchestra.run.smithi111.stdout: librabbitmq-0.9.0-3.el8.x86_64 2023-10-11T10:38:33.044 INFO:teuthology.orchestra.run.smithi111.stdout: libradosstriper1-2:18.0.0-6661.g4bf94f59.el8.x86_64 2023-10-11T10:38:33.044 INFO:teuthology.orchestra.run.smithi111.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-10-11T10:38:33.045 INFO:teuthology.orchestra.run.smithi111.stdout: librgw2-2:18.0.0-6661.g4bf94f59.el8.x86_64 2023-10-11T10:38:33.045 INFO:teuthology.orchestra.run.smithi111.stdout: libunwind-1.3.1-3.el8.x86_64 2023-10-11T10:38:33.045 INFO:teuthology.orchestra.run.smithi111.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-10-11T10:38:33.045 INFO:teuthology.orchestra.run.smithi111.stdout: python3-ceph-argparse-2:18.0.0-6661.g4bf94f59.el8.x86_64 2023-10-11T10:38:33.045 INFO:teuthology.orchestra.run.smithi111.stdout: python3-ceph-common-2:18.0.0-6661.g4bf94f59.el8.x86_64 2023-10-11T10:38:33.046 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cephfs-2:18.0.0-6661.g4bf94f59.el8.x86_64 2023-10-11T10:38:33.046 INFO:teuthology.orchestra.run.smithi111.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-10-11T10:38:33.046 INFO:teuthology.orchestra.run.smithi111.stdout: python3-rados-2:18.0.0-6661.g4bf94f59.el8.x86_64 2023-10-11T10:38:33.046 INFO:teuthology.orchestra.run.smithi111.stdout: python3-rbd-2:18.0.0-6661.g4bf94f59.el8.x86_64 2023-10-11T10:38:33.047 INFO:teuthology.orchestra.run.smithi111.stdout: python3-rgw-2:18.0.0-6661.g4bf94f59.el8.x86_64 2023-10-11T10:38:33.047 INFO:teuthology.orchestra.run.smithi111.stdout: thrift-0.13.0-2.el8.x86_64 2023-10-11T10:38:33.047 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T10:38:33.047 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2023-10-11T10:38:33.077 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2023-10-11T10:38:33.077 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-base-2:18.0.0-6661.g4bf94f59.el8.x86_64 1/25 2023-10-11T10:38:33.078 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-common-2:18.0.0-6661.g4bf94f59.el8.x86_64 2/25 2023-10-11T10:38:33.078 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-radosgw-2:18.0.0-6661.g4bf94f59.el8.x86_64 3/25 2023-10-11T10:38:33.078 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-selinux-2:18.0.0-6661.g4bf94f59.el8.x86_64 4/25 2023-10-11T10:38:33.078 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : libcephfs2-2:18.0.0-6661.g4bf94f59.el8.x86_64 5/25 2023-10-11T10:38:33.078 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : libradosstriper1-2:18.0.0-6661.g4bf94f59.el8.x86_6 6/25 2023-10-11T10:38:33.079 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : librgw2-2:18.0.0-6661.g4bf94f59.el8.x86_64 7/25 2023-10-11T10:38:33.079 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-ceph-argparse-2:18.0.0-6661.g4bf94f59.el8. 8/25 2023-10-11T10:38:33.079 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-ceph-common-2:18.0.0-6661.g4bf94f59.el8.x8 9/25 2023-10-11T10:38:33.079 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-cephfs-2:18.0.0-6661.g4bf94f59.el8.x86_64 10/25 2023-10-11T10:38:33.079 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-rados-2:18.0.0-6661.g4bf94f59.el8.x86_64 11/25 2023-10-11T10:38:33.079 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-rbd-2:18.0.0-6661.g4bf94f59.el8.x86_64 12/25 2023-10-11T10:38:33.080 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-rgw-2:18.0.0-6661.g4bf94f59.el8.x86_64 13/25 2023-10-11T10:38:33.080 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 14/25 2023-10-11T10:38:33.080 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 15/25 2023-10-11T10:38:33.080 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 16/25 2023-10-11T10:38:33.080 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 17/25 2023-10-11T10:38:33.081 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 18/25 2023-10-11T10:38:33.081 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 19/25 2023-10-11T10:38:33.081 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 20/25 2023-10-11T10:38:33.081 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 21/25 2023-10-11T10:38:33.081 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : librados2-2:18.0.0-6661.g4bf94f59.el8.x86_64 22/25 2023-10-11T10:38:33.082 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2023-10-11T10:38:33.082 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : librbd1-2:18.0.0-6661.g4bf94f59.el8.x86_64 24/25 2023-10-11T10:38:33.225 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install ceph-test 2023-10-11T10:38:33.572 INFO:teuthology.orchestra.run.smithi111.stdout:Updating Subscription Management repositories. 2023-10-11T10:38:34.372 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:00:55 ago on Wed 11 Oct 2023 10:37:39 AM UTC. 2023-10-11T10:38:34.491 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2023-10-11T10:38:34.491 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:00:53 ago on Wed 11 Oct 2023 10:37:41 AM UTC. 2023-10-11T10:38:35.825 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2023-10-11T10:38:35.826 INFO:teuthology.orchestra.run.smithi111.stdout:============================================================================================ 2023-10-11T10:38:35.826 INFO:teuthology.orchestra.run.smithi111.stdout: Package Arch Version Repository Size 2023-10-11T10:38:35.827 INFO:teuthology.orchestra.run.smithi111.stdout:============================================================================================ 2023-10-11T10:38:35.827 INFO:teuthology.orchestra.run.smithi111.stdout:Installing: 2023-10-11T10:38:35.827 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-test x86_64 2:18.0.0-6661.g4bf94f59.el8 ceph 49 M 2023-10-11T10:38:35.827 INFO:teuthology.orchestra.run.smithi111.stdout:Installing dependencies: 2023-10-11T10:38:35.827 INFO:teuthology.orchestra.run.smithi111.stdout: jq x86_64 1.6-3.el8 rhel-8-for-x86_64-appstream-rpms 202 k 2023-10-11T10:38:35.827 INFO:teuthology.orchestra.run.smithi111.stdout: libcephsqlite x86_64 2:18.0.0-6661.g4bf94f59.el8 ceph 184 k 2023-10-11T10:38:35.828 INFO:teuthology.orchestra.run.smithi111.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2023-10-11T10:38:35.828 INFO:teuthology.orchestra.run.smithi111.stdout: socat x86_64 1.7.4.1-1.el8 rhel-8-for-x86_64-appstream-rpms 323 k 2023-10-11T10:38:35.828 INFO:teuthology.orchestra.run.smithi111.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2023-10-11T10:38:35.828 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T10:38:35.828 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2023-10-11T10:38:35.829 INFO:teuthology.orchestra.run.smithi111.stdout:============================================================================================ 2023-10-11T10:38:35.829 INFO:teuthology.orchestra.run.smithi111.stdout:Install 6 Packages 2023-10-11T10:38:35.829 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T10:38:35.830 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 50 M 2023-10-11T10:38:35.830 INFO:teuthology.orchestra.run.smithi111.stdout:Installed size: 222 M 2023-10-11T10:38:35.830 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2023-10-11T10:38:35.859 INFO:teuthology.orchestra.run.smithi111.stdout:(1/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 2.2 MB/s | 70 kB 00:00 2023-10-11T10:38:35.935 INFO:teuthology.orchestra.run.smithi111.stdout:(2/6): socat-1.7.4.1-1.el8.x86_64.rpm 4.2 MB/s | 323 kB 00:00 2023-10-11T10:38:35.993 INFO:teuthology.orchestra.run.smithi111.stdout:(3/6): oniguruma-6.8.2-2.el8.x86_64.rpm 3.2 MB/s | 187 kB 00:00 2023-10-11T10:38:36.044 INFO:teuthology.orchestra.run.smithi111.stdout:(4/6): jq-1.6-3.el8.x86_64.rpm 4.0 MB/s | 202 kB 00:00 2023-10-11T10:38:36.195 INFO:teuthology.orchestra.run.smithi111.stdout:(5/6): libcephsqlite-18.0.0-6661.g4bf94f59.el8. 502 kB/s | 184 kB 00:00 2023-10-11T10:38:37.196 INFO:teuthology.orchestra.run.smithi050.stdout:Installed products updated. 2023-10-11T10:38:37.420 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-11T10:38:37.420 INFO:teuthology.orchestra.run.smithi050.stdout:Upgraded: 2023-10-11T10:38:37.420 INFO:teuthology.orchestra.run.smithi050.stdout: librados2-2:18.0.0-6661.g4bf94f59.el8.x86_64 2023-10-11T10:38:37.421 INFO:teuthology.orchestra.run.smithi050.stdout: librbd1-2:18.0.0-6661.g4bf94f59.el8.x86_64 2023-10-11T10:38:37.421 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2023-10-11T10:38:37.421 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-base-2:18.0.0-6661.g4bf94f59.el8.x86_64 2023-10-11T10:38:37.421 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-common-2:18.0.0-6661.g4bf94f59.el8.x86_64 2023-10-11T10:38:37.421 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-radosgw-2:18.0.0-6661.g4bf94f59.el8.x86_64 2023-10-11T10:38:37.421 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-selinux-2:18.0.0-6661.g4bf94f59.el8.x86_64 2023-10-11T10:38:37.422 INFO:teuthology.orchestra.run.smithi050.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-10-11T10:38:37.422 INFO:teuthology.orchestra.run.smithi050.stdout: libcephfs2-2:18.0.0-6661.g4bf94f59.el8.x86_64 2023-10-11T10:38:37.422 INFO:teuthology.orchestra.run.smithi050.stdout: liboath-2.6.2-3.el8.x86_64 2023-10-11T10:38:37.422 INFO:teuthology.orchestra.run.smithi050.stdout: librabbitmq-0.9.0-3.el8.x86_64 2023-10-11T10:38:37.422 INFO:teuthology.orchestra.run.smithi050.stdout: libradosstriper1-2:18.0.0-6661.g4bf94f59.el8.x86_64 2023-10-11T10:38:37.423 INFO:teuthology.orchestra.run.smithi050.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-10-11T10:38:37.423 INFO:teuthology.orchestra.run.smithi050.stdout: librgw2-2:18.0.0-6661.g4bf94f59.el8.x86_64 2023-10-11T10:38:37.423 INFO:teuthology.orchestra.run.smithi050.stdout: libunwind-1.3.1-3.el8.x86_64 2023-10-11T10:38:37.423 INFO:teuthology.orchestra.run.smithi050.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-10-11T10:38:37.423 INFO:teuthology.orchestra.run.smithi050.stdout: python3-ceph-argparse-2:18.0.0-6661.g4bf94f59.el8.x86_64 2023-10-11T10:38:37.423 INFO:teuthology.orchestra.run.smithi050.stdout: python3-ceph-common-2:18.0.0-6661.g4bf94f59.el8.x86_64 2023-10-11T10:38:37.424 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cephfs-2:18.0.0-6661.g4bf94f59.el8.x86_64 2023-10-11T10:38:37.424 INFO:teuthology.orchestra.run.smithi050.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-10-11T10:38:37.424 INFO:teuthology.orchestra.run.smithi050.stdout: python3-rados-2:18.0.0-6661.g4bf94f59.el8.x86_64 2023-10-11T10:38:37.424 INFO:teuthology.orchestra.run.smithi050.stdout: python3-rbd-2:18.0.0-6661.g4bf94f59.el8.x86_64 2023-10-11T10:38:37.424 INFO:teuthology.orchestra.run.smithi050.stdout: python3-rgw-2:18.0.0-6661.g4bf94f59.el8.x86_64 2023-10-11T10:38:37.424 INFO:teuthology.orchestra.run.smithi050.stdout: thrift-0.13.0-2.el8.x86_64 2023-10-11T10:38:37.425 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-11T10:38:37.425 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2023-10-11T10:38:37.629 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install ceph-test 2023-10-11T10:38:37.999 INFO:teuthology.orchestra.run.smithi050.stdout:Updating Subscription Management repositories. 2023-10-11T10:38:38.386 INFO:teuthology.orchestra.run.smithi111.stdout:(6/6): ceph-test-18.0.0-6661.g4bf94f59.el8.x86_ 19 MB/s | 49 MB 00:02 2023-10-11T10:38:38.388 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2023-10-11T10:38:38.388 INFO:teuthology.orchestra.run.smithi111.stdout:Total 19 MB/s | 50 MB 00:02 2023-10-11T10:38:38.426 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2023-10-11T10:38:38.448 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2023-10-11T10:38:38.449 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2023-10-11T10:38:38.717 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2023-10-11T10:38:38.721 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2023-10-11T10:38:38.912 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:00:57 ago on Wed 11 Oct 2023 10:37:41 AM UTC. 2023-10-11T10:38:39.168 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2023-10-11T10:38:39.329 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 1/6 2023-10-11T10:38:39.422 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 1/6 2023-10-11T10:38:39.657 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : jq-1.6-3.el8.x86_64 2/6 2023-10-11T10:38:39.812 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 3/6 2023-10-11T10:38:40.002 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 4/6 2023-10-11T10:38:40.229 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : libcephsqlite-2:18.0.0-6661.g4bf94f59.el8.x86_64 5/6 2023-10-11T10:38:40.446 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2023-10-11T10:38:40.447 INFO:teuthology.orchestra.run.smithi050.stdout:============================================================================================ 2023-10-11T10:38:40.448 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repository Size 2023-10-11T10:38:40.448 INFO:teuthology.orchestra.run.smithi050.stdout:============================================================================================ 2023-10-11T10:38:40.448 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2023-10-11T10:38:40.448 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-test x86_64 2:18.0.0-6661.g4bf94f59.el8 ceph 49 M 2023-10-11T10:38:40.448 INFO:teuthology.orchestra.run.smithi050.stdout:Installing dependencies: 2023-10-11T10:38:40.449 INFO:teuthology.orchestra.run.smithi050.stdout: jq x86_64 1.6-3.el8 rhel-8-for-x86_64-appstream-rpms 202 k 2023-10-11T10:38:40.449 INFO:teuthology.orchestra.run.smithi050.stdout: libcephsqlite x86_64 2:18.0.0-6661.g4bf94f59.el8 ceph 184 k 2023-10-11T10:38:40.449 INFO:teuthology.orchestra.run.smithi050.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2023-10-11T10:38:40.449 INFO:teuthology.orchestra.run.smithi050.stdout: socat x86_64 1.7.4.1-1.el8 rhel-8-for-x86_64-appstream-rpms 323 k 2023-10-11T10:38:40.449 INFO:teuthology.orchestra.run.smithi050.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2023-10-11T10:38:40.450 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-11T10:38:40.450 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2023-10-11T10:38:40.450 INFO:teuthology.orchestra.run.smithi050.stdout:============================================================================================ 2023-10-11T10:38:40.450 INFO:teuthology.orchestra.run.smithi050.stdout:Install 6 Packages 2023-10-11T10:38:40.450 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-11T10:38:40.451 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 50 M 2023-10-11T10:38:40.451 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 222 M 2023-10-11T10:38:40.451 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2023-10-11T10:38:40.483 INFO:teuthology.orchestra.run.smithi050.stdout:(1/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 2.1 MB/s | 70 kB 00:00 2023-10-11T10:38:40.558 INFO:teuthology.orchestra.run.smithi050.stdout:(2/6): socat-1.7.4.1-1.el8.x86_64.rpm 4.8 MB/s | 323 kB 00:00 2023-10-11T10:38:40.600 INFO:teuthology.orchestra.run.smithi050.stdout:(3/6): oniguruma-6.8.2-2.el8.x86_64.rpm 4.4 MB/s | 187 kB 00:00 2023-10-11T10:38:40.642 INFO:teuthology.orchestra.run.smithi050.stdout:(4/6): jq-1.6-3.el8.x86_64.rpm 4.7 MB/s | 202 kB 00:00 2023-10-11T10:38:40.734 INFO:teuthology.orchestra.run.smithi050.stdout:(5/6): libcephsqlite-18.0.0-6661.g4bf94f59.el8. 647 kB/s | 184 kB 00:00 2023-10-11T10:38:42.826 INFO:teuthology.orchestra.run.smithi050.stdout:(6/6): ceph-test-18.0.0-6661.g4bf94f59.el8.x86_ 21 MB/s | 49 MB 00:02 2023-10-11T10:38:42.828 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2023-10-11T10:38:42.828 INFO:teuthology.orchestra.run.smithi050.stdout:Total 21 MB/s | 50 MB 00:02 2023-10-11T10:38:42.867 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2023-10-11T10:38:42.891 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2023-10-11T10:38:42.891 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2023-10-11T10:38:43.176 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2023-10-11T10:38:43.179 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2023-10-11T10:38:43.669 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2023-10-11T10:38:43.797 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 1/6 2023-10-11T10:38:43.880 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 1/6 2023-10-11T10:38:44.070 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : jq-1.6-3.el8.x86_64 2/6 2023-10-11T10:38:44.257 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 3/6 2023-10-11T10:38:44.414 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 4/6 2023-10-11T10:38:44.597 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : libcephsqlite-2:18.0.0-6661.g4bf94f59.el8.x86_64 5/6 2023-10-11T10:38:44.916 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: libcephsqlite-2:18.0.0-6661.g4bf94f59.el8.x86_64 5/6 2023-10-11T10:38:45.175 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-test-2:18.0.0-6661.g4bf94f59.el8.x86_64 6/6 2023-10-11T10:38:45.400 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-test-2:18.0.0-6661.g4bf94f59.el8.x86_64 6/6 2023-10-11T10:38:45.400 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-test-2:18.0.0-6661.g4bf94f59.el8.x86_64 1/6 2023-10-11T10:38:45.401 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : libcephsqlite-2:18.0.0-6661.g4bf94f59.el8.x86_64 2/6 2023-10-11T10:38:45.401 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 3/6 2023-10-11T10:38:45.401 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 4/6 2023-10-11T10:38:45.401 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 5/6 2023-10-11T10:38:46.481 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : jq-1.6-3.el8.x86_64 6/6 2023-10-11T10:38:46.481 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:01:07 ago on Wed 11 Oct 2023 10:37:39 AM UTC. 2023-10-11T10:38:49.254 INFO:teuthology.orchestra.run.smithi111.stdout:Installed products updated. 2023-10-11T10:38:49.437 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T10:38:49.437 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2023-10-11T10:38:49.438 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-test-2:18.0.0-6661.g4bf94f59.el8.x86_64 2023-10-11T10:38:49.438 INFO:teuthology.orchestra.run.smithi111.stdout: jq-1.6-3.el8.x86_64 2023-10-11T10:38:49.438 INFO:teuthology.orchestra.run.smithi111.stdout: libcephsqlite-2:18.0.0-6661.g4bf94f59.el8.x86_64 2023-10-11T10:38:49.438 INFO:teuthology.orchestra.run.smithi111.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-10-11T10:38:49.438 INFO:teuthology.orchestra.run.smithi111.stdout: socat-1.7.4.1-1.el8.x86_64 2023-10-11T10:38:49.439 INFO:teuthology.orchestra.run.smithi111.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-10-11T10:38:49.439 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T10:38:49.439 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2023-10-11T10:38:49.597 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install ceph 2023-10-11T10:38:49.615 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: libcephsqlite-2:18.0.0-6661.g4bf94f59.el8.x86_64 5/6 2023-10-11T10:38:49.828 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-test-2:18.0.0-6661.g4bf94f59.el8.x86_64 6/6 2023-10-11T10:38:49.931 INFO:teuthology.orchestra.run.smithi111.stdout:Updating Subscription Management repositories. 2023-10-11T10:38:50.065 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-test-2:18.0.0-6661.g4bf94f59.el8.x86_64 6/6 2023-10-11T10:38:50.065 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-test-2:18.0.0-6661.g4bf94f59.el8.x86_64 1/6 2023-10-11T10:38:50.065 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : libcephsqlite-2:18.0.0-6661.g4bf94f59.el8.x86_64 2/6 2023-10-11T10:38:50.065 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 3/6 2023-10-11T10:38:50.066 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 4/6 2023-10-11T10:38:50.066 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 5/6 2023-10-11T10:38:50.802 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:01:11 ago on Wed 11 Oct 2023 10:37:39 AM UTC. 2023-10-11T10:38:51.171 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : jq-1.6-3.el8.x86_64 6/6 2023-10-11T10:38:51.171 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:01:10 ago on Wed 11 Oct 2023 10:37:41 AM UTC. 2023-10-11T10:38:52.199 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2023-10-11T10:38:52.202 INFO:teuthology.orchestra.run.smithi111.stdout:======================================================================================================= 2023-10-11T10:38:52.203 INFO:teuthology.orchestra.run.smithi111.stdout: Package Arch Version Repository Size 2023-10-11T10:38:52.203 INFO:teuthology.orchestra.run.smithi111.stdout:======================================================================================================= 2023-10-11T10:38:52.203 INFO:teuthology.orchestra.run.smithi111.stdout:Installing: 2023-10-11T10:38:52.203 INFO:teuthology.orchestra.run.smithi111.stdout: ceph x86_64 2:18.0.0-6661.g4bf94f59.el8 ceph 6.4 k 2023-10-11T10:38:52.204 INFO:teuthology.orchestra.run.smithi111.stdout:Installing dependencies: 2023-10-11T10:38:52.204 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mds x86_64 2:18.0.0-6661.g4bf94f59.el8 ceph 2.3 M 2023-10-11T10:38:52.204 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mgr x86_64 2:18.0.0-6661.g4bf94f59.el8 ceph 1.6 M 2023-10-11T10:38:52.204 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mgr-modules-core noarch 2:18.0.0-6661.g4bf94f59.el8 ceph-noarch 242 k 2023-10-11T10:38:52.205 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mon x86_64 2:18.0.0-6661.g4bf94f59.el8 ceph 4.6 M 2023-10-11T10:38:52.205 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-osd x86_64 2:18.0.0-6661.g4bf94f59.el8 ceph 18 M 2023-10-11T10:38:52.205 INFO:teuthology.orchestra.run.smithi111.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-10-11T10:38:52.205 INFO:teuthology.orchestra.run.smithi111.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-10-11T10:38:52.206 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cffi x86_64 1.11.5-5.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2023-10-11T10:38:52.206 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-10-11T10:38:52.206 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-10-11T10:38:52.206 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cryptography x86_64 3.2.1-5.el8 rhel-8-for-x86_64-baseos-rpms 559 k 2023-10-11T10:38:52.207 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-10-11T10:38:52.207 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-10-11T10:38:52.207 INFO:teuthology.orchestra.run.smithi111.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-10-11T10:38:52.207 INFO:teuthology.orchestra.run.smithi111.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2023-10-11T10:38:52.208 INFO:teuthology.orchestra.run.smithi111.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2023-10-11T10:38:52.208 INFO:teuthology.orchestra.run.smithi111.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-10-11T10:38:52.208 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-10-11T10:38:52.208 INFO:teuthology.orchestra.run.smithi111.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-10-11T10:38:52.209 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2023-10-11T10:38:52.209 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2023-10-11T10:38:52.209 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2023-10-11T10:38:52.209 INFO:teuthology.orchestra.run.smithi111.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-10-11T10:38:52.210 INFO:teuthology.orchestra.run.smithi111.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-10-11T10:38:52.210 INFO:teuthology.orchestra.run.smithi111.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-10-11T10:38:52.210 INFO:teuthology.orchestra.run.smithi111.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-10-11T10:38:52.211 INFO:teuthology.orchestra.run.smithi111.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-10-11T10:38:52.211 INFO:teuthology.orchestra.run.smithi111.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-10-11T10:38:52.211 INFO:teuthology.orchestra.run.smithi111.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-10-11T10:38:52.211 INFO:teuthology.orchestra.run.smithi111.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2023-10-11T10:38:52.212 INFO:teuthology.orchestra.run.smithi111.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-10-11T10:38:52.212 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T10:38:52.212 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2023-10-11T10:38:52.212 INFO:teuthology.orchestra.run.smithi111.stdout:======================================================================================================= 2023-10-11T10:38:52.212 INFO:teuthology.orchestra.run.smithi111.stdout:Install 32 Packages 2023-10-11T10:38:52.212 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T10:38:52.213 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 30 M 2023-10-11T10:38:52.213 INFO:teuthology.orchestra.run.smithi111.stdout:Installed size: 121 M 2023-10-11T10:38:52.214 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2023-10-11T10:38:52.805 INFO:teuthology.orchestra.run.smithi111.stdout:(1/32): ceph-18.0.0-6661.g4bf94f59.el8.x86_64.r 16 kB/s | 6.4 kB 00:00 2023-10-11T10:38:53.155 INFO:teuthology.orchestra.run.smithi111.stdout:(2/32): ceph-mgr-18.0.0-6661.g4bf94f59.el8.x86_ 2.1 MB/s | 1.6 MB 00:00 2023-10-11T10:38:53.464 INFO:teuthology.orchestra.run.smithi111.stdout:(3/32): ceph-mds-18.0.0-6661.g4bf94f59.el8.x86_ 2.2 MB/s | 2.3 MB 00:01 2023-10-11T10:38:53.564 INFO:teuthology.orchestra.run.smithi111.stdout:(4/32): ceph-mgr-modules-core-18.0.0-6661.g4bf9 2.4 MB/s | 242 kB 00:00 2023-10-11T10:38:53.614 INFO:teuthology.orchestra.run.smithi111.stdout:(5/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rpm 877 kB/s | 44 kB 00:00 2023-10-11T10:38:53.665 INFO:teuthology.orchestra.run.smithi111.stdout:(6/32): python3-beautifulsoup4-4.6.3-2.el8.1.no 3.6 MB/s | 185 kB 00:00 2023-10-11T10:38:53.723 INFO:teuthology.orchestra.run.smithi111.stdout:(7/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 2.9 MB/s | 173 kB 00:00 2023-10-11T10:38:53.765 INFO:teuthology.orchestra.run.smithi111.stdout:(8/32): python3-cherrypy-18.4.0-1.el8.noarch.rp 9.1 MB/s | 384 kB 00:00 2023-10-11T10:38:53.790 INFO:teuthology.orchestra.run.smithi111.stdout:(9/32): python3-jaraco-6.2-6.el8.noarch.rpm 443 kB/s | 11 kB 00:00 2023-10-11T10:38:53.824 INFO:teuthology.orchestra.run.smithi111.stdout:(10/32): python3-jaraco-functools-2.0-4.el8.noa 523 kB/s | 18 kB 00:00 2023-10-11T10:38:53.858 INFO:teuthology.orchestra.run.smithi111.stdout:(11/32): python3-logutils-0.3.5-11.el8.noarch.r 1.4 MB/s | 49 kB 00:00 2023-10-11T10:38:53.892 INFO:teuthology.orchestra.run.smithi111.stdout:(12/32): python3-more-itertools-7.2.0-3.el8.noa 1.7 MB/s | 59 kB 00:00 2023-10-11T10:38:53.925 INFO:teuthology.orchestra.run.smithi111.stdout:(13/32): python3-pecan-1.3.2-9.el8.noarch.rpm 8.3 MB/s | 283 kB 00:00 2023-10-11T10:38:53.959 INFO:teuthology.orchestra.run.smithi111.stdout:(14/32): python3-portend-2.6-1.el8.noarch.rpm 469 kB/s | 16 kB 00:00 2023-10-11T10:38:54.026 INFO:teuthology.orchestra.run.smithi111.stdout:(15/32): python3-simplegeneric-0.8.1-17.el8.noa 282 kB/s | 19 kB 00:00 2023-10-11T10:38:54.087 INFO:teuthology.orchestra.run.smithi050.stdout:Installed products updated. 2023-10-11T10:38:54.202 INFO:teuthology.orchestra.run.smithi111.stdout:(16/32): ceph-osd-18.0.0-6661.g4bf94f59.el8.x86 17 MB/s | 18 MB 00:01 2023-10-11T10:38:54.227 INFO:teuthology.orchestra.run.smithi111.stdout:(17/32): python3-singledispatch-3.4.0.3-18.el8. 119 kB/s | 24 kB 00:00 2023-10-11T10:38:54.287 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-11T10:38:54.287 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2023-10-11T10:38:54.287 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-test-2:18.0.0-6661.g4bf94f59.el8.x86_64 2023-10-11T10:38:54.287 INFO:teuthology.orchestra.run.smithi050.stdout: jq-1.6-3.el8.x86_64 2023-10-11T10:38:54.288 INFO:teuthology.orchestra.run.smithi050.stdout: libcephsqlite-2:18.0.0-6661.g4bf94f59.el8.x86_64 2023-10-11T10:38:54.288 INFO:teuthology.orchestra.run.smithi050.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-10-11T10:38:54.288 INFO:teuthology.orchestra.run.smithi050.stdout: socat-1.7.4.1-1.el8.x86_64 2023-10-11T10:38:54.288 INFO:teuthology.orchestra.run.smithi050.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-10-11T10:38:54.288 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-11T10:38:54.289 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2023-10-11T10:38:54.327 INFO:teuthology.orchestra.run.smithi111.stdout:(18/32): ceph-mon-18.0.0-6661.g4bf94f59.el8.x86 3.0 MB/s | 4.6 MB 00:01 2023-10-11T10:38:54.352 INFO:teuthology.orchestra.run.smithi111.stdout:(19/32): python3-tempora-1.14.1-5.el8.noarch.rp 194 kB/s | 29 kB 00:00 2023-10-11T10:38:54.377 INFO:teuthology.orchestra.run.smithi111.stdout:(20/32): python3-trustme-0.6.0-4.el8.noarch.rpm 183 kB/s | 27 kB 00:00 2023-10-11T10:38:54.411 INFO:teuthology.orchestra.run.smithi111.stdout:(21/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 7.4 MB/s | 251 kB 00:00 2023-10-11T10:38:54.433 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install ceph 2023-10-11T10:38:54.436 INFO:teuthology.orchestra.run.smithi111.stdout:(22/32): python3-webtest-2.0.33-1.el8.noarch.rp 1.4 MB/s | 85 kB 00:00 2023-10-11T10:38:54.462 INFO:teuthology.orchestra.run.smithi111.stdout:(23/32): python3-zc-lockfile-2.0-2.el8.noarch.r 458 kB/s | 23 kB 00:00 2023-10-11T10:38:54.487 INFO:teuthology.orchestra.run.smithi111.stdout:(24/32): python3-waitress-1.4.3-1.el8.noarch.rp 1.6 MB/s | 255 kB 00:00 2023-10-11T10:38:54.599 INFO:teuthology.orchestra.run.smithi111.stdout:(25/32): python3-cffi-1.11.5-5.el8.x86_64.rpm 2.2 MB/s | 238 kB 00:00 2023-10-11T10:38:54.629 INFO:teuthology.orchestra.run.smithi111.stdout:(26/32): python3-pycparser-2.14-14.el8.noarch.r 567 kB/s | 109 kB 00:00 2023-10-11T10:38:54.663 INFO:teuthology.orchestra.run.smithi111.stdout:(27/32): python3-cryptography-3.2.1-5.el8.x86_6 2.7 MB/s | 559 kB 00:00 2023-10-11T10:38:54.705 INFO:teuthology.orchestra.run.smithi111.stdout:(28/32): python3-pytz-2017.2-9.el8.noarch.rpm 515 kB/s | 54 kB 00:00 2023-10-11T10:38:54.730 INFO:teuthology.orchestra.run.smithi111.stdout:(29/32): python3-markupsafe-0.23-19.el8.x86_64. 386 kB/s | 39 kB 00:00 2023-10-11T10:38:54.788 INFO:teuthology.orchestra.run.smithi050.stdout:Updating Subscription Management repositories. 2023-10-11T10:38:54.789 INFO:teuthology.orchestra.run.smithi111.stdout:(30/32): python3-mako-1.0.6-13.el8.noarch.rpm 1.2 MB/s | 157 kB 00:00 2023-10-11T10:38:54.822 INFO:teuthology.orchestra.run.smithi111.stdout:(31/32): python3-werkzeug-0.12.2-4.el8.noarch.r 3.8 MB/s | 457 kB 00:00 2023-10-11T10:38:54.855 INFO:teuthology.orchestra.run.smithi111.stdout:(32/32): python3-pyOpenSSL-19.0.0-1.el8.noarch. 827 kB/s | 103 kB 00:00 2023-10-11T10:38:54.862 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2023-10-11T10:38:54.862 INFO:teuthology.orchestra.run.smithi111.stdout:Total 11 MB/s | 30 MB 00:02 2023-10-11T10:38:54.956 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2023-10-11T10:38:54.998 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2023-10-11T10:38:54.998 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2023-10-11T10:38:55.209 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2023-10-11T10:38:55.213 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2023-10-11T10:38:56.120 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:01:15 ago on Wed 11 Oct 2023 10:37:41 AM UTC. 2023-10-11T10:38:56.381 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2023-10-11T10:38:56.598 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2023-10-11T10:38:57.001 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2023-10-11T10:38:57.196 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 3/32 2023-10-11T10:38:57.452 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 4/32 2023-10-11T10:38:57.663 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2023-10-11T10:38:57.666 INFO:teuthology.orchestra.run.smithi050.stdout:======================================================================================================= 2023-10-11T10:38:57.667 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repository Size 2023-10-11T10:38:57.667 INFO:teuthology.orchestra.run.smithi050.stdout:======================================================================================================= 2023-10-11T10:38:57.667 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2023-10-11T10:38:57.667 INFO:teuthology.orchestra.run.smithi050.stdout: ceph x86_64 2:18.0.0-6661.g4bf94f59.el8 ceph 6.4 k 2023-10-11T10:38:57.667 INFO:teuthology.orchestra.run.smithi050.stdout:Installing dependencies: 2023-10-11T10:38:57.667 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mds x86_64 2:18.0.0-6661.g4bf94f59.el8 ceph 2.3 M 2023-10-11T10:38:57.668 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr x86_64 2:18.0.0-6661.g4bf94f59.el8 ceph 1.6 M 2023-10-11T10:38:57.668 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-modules-core noarch 2:18.0.0-6661.g4bf94f59.el8 ceph-noarch 242 k 2023-10-11T10:38:57.668 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mon x86_64 2:18.0.0-6661.g4bf94f59.el8 ceph 4.6 M 2023-10-11T10:38:57.668 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-osd x86_64 2:18.0.0-6661.g4bf94f59.el8 ceph 18 M 2023-10-11T10:38:57.669 INFO:teuthology.orchestra.run.smithi050.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-10-11T10:38:57.669 INFO:teuthology.orchestra.run.smithi050.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-10-11T10:38:57.669 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cffi x86_64 1.11.5-5.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2023-10-11T10:38:57.669 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-10-11T10:38:57.669 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-10-11T10:38:57.670 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cryptography x86_64 3.2.1-5.el8 rhel-8-for-x86_64-baseos-rpms 559 k 2023-10-11T10:38:57.670 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-10-11T10:38:57.670 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-10-11T10:38:57.670 INFO:teuthology.orchestra.run.smithi050.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-10-11T10:38:57.670 INFO:teuthology.orchestra.run.smithi050.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2023-10-11T10:38:57.671 INFO:teuthology.orchestra.run.smithi050.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2023-10-11T10:38:57.671 INFO:teuthology.orchestra.run.smithi050.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-10-11T10:38:57.671 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-10-11T10:38:57.671 INFO:teuthology.orchestra.run.smithi050.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-10-11T10:38:57.671 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2023-10-11T10:38:57.671 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2023-10-11T10:38:57.672 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2023-10-11T10:38:57.672 INFO:teuthology.orchestra.run.smithi050.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-10-11T10:38:57.672 INFO:teuthology.orchestra.run.smithi050.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-10-11T10:38:57.672 INFO:teuthology.orchestra.run.smithi050.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-10-11T10:38:57.672 INFO:teuthology.orchestra.run.smithi050.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-10-11T10:38:57.673 INFO:teuthology.orchestra.run.smithi050.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-10-11T10:38:57.673 INFO:teuthology.orchestra.run.smithi050.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-10-11T10:38:57.673 INFO:teuthology.orchestra.run.smithi050.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-10-11T10:38:57.673 INFO:teuthology.orchestra.run.smithi050.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2023-10-11T10:38:57.673 INFO:teuthology.orchestra.run.smithi050.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-10-11T10:38:57.674 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-11T10:38:57.674 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2023-10-11T10:38:57.674 INFO:teuthology.orchestra.run.smithi050.stdout:======================================================================================================= 2023-10-11T10:38:57.674 INFO:teuthology.orchestra.run.smithi050.stdout:Install 32 Packages 2023-10-11T10:38:57.674 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-11T10:38:57.675 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 30 M 2023-10-11T10:38:57.675 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 121 M 2023-10-11T10:38:57.676 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2023-10-11T10:38:57.701 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 5/32 2023-10-11T10:38:57.887 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 6/32 2023-10-11T10:38:58.136 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 7/32 2023-10-11T10:38:58.137 INFO:teuthology.orchestra.run.smithi050.stdout:(1/32): ceph-18.0.0-6661.g4bf94f59.el8.x86_64.r 20 kB/s | 6.4 kB 00:00 2023-10-11T10:38:58.539 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 8/32 2023-10-11T10:38:58.820 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-cryptography-3.2.1-5.el8.x86_64 9/32 2023-10-11T10:38:58.853 INFO:teuthology.orchestra.run.smithi050.stdout:(2/32): ceph-mds-18.0.0-6661.g4bf94f59.el8.x86_ 2.2 MB/s | 2.3 MB 00:01 2023-10-11T10:38:58.983 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 10/32 2023-10-11T10:38:59.177 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 11/32 2023-10-11T10:38:59.359 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 12/32 2023-10-11T10:38:59.370 INFO:teuthology.orchestra.run.smithi050.stdout:(3/32): ceph-mgr-18.0.0-6661.g4bf94f59.el8.x86_ 1.0 MB/s | 1.6 MB 00:01 2023-10-11T10:38:59.487 INFO:teuthology.orchestra.run.smithi050.stdout:(4/32): ceph-mgr-modules-core-18.0.0-6661.g4bf9 2.0 MB/s | 242 kB 00:00 2023-10-11T10:38:59.602 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 13/32 2023-10-11T10:38:59.654 INFO:teuthology.orchestra.run.smithi050.stdout:(5/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rpm 262 kB/s | 44 kB 00:00 2023-10-11T10:38:59.730 INFO:teuthology.orchestra.run.smithi050.stdout:(6/32): python3-beautifulsoup4-4.6.3-2.el8.1.no 2.4 MB/s | 185 kB 00:00 2023-10-11T10:38:59.763 INFO:teuthology.orchestra.run.smithi050.stdout:(7/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 5.1 MB/s | 173 kB 00:00 2023-10-11T10:38:59.816 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 14/32 2023-10-11T10:38:59.822 INFO:teuthology.orchestra.run.smithi050.stdout:(8/32): python3-cherrypy-18.4.0-1.el8.noarch.rp 6.4 MB/s | 384 kB 00:00 2023-10-11T10:38:59.864 INFO:teuthology.orchestra.run.smithi050.stdout:(9/32): python3-jaraco-6.2-6.el8.noarch.rpm 267 kB/s | 11 kB 00:00 2023-10-11T10:38:59.905 INFO:teuthology.orchestra.run.smithi050.stdout:(10/32): python3-jaraco-functools-2.0-4.el8.noa 425 kB/s | 18 kB 00:00 2023-10-11T10:38:59.948 INFO:teuthology.orchestra.run.smithi050.stdout:(11/32): python3-logutils-0.3.5-11.el8.noarch.r 1.1 MB/s | 49 kB 00:00 2023-10-11T10:38:59.971 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 15/32 2023-10-11T10:39:00.132 INFO:teuthology.orchestra.run.smithi050.stdout:(12/32): ceph-osd-18.0.0-6661.g4bf94f59.el8.x86 14 MB/s | 18 MB 00:01 2023-10-11T10:39:00.158 INFO:teuthology.orchestra.run.smithi050.stdout:(13/32): python3-more-itertools-7.2.0-3.el8.noa 282 kB/s | 59 kB 00:00 2023-10-11T10:39:00.161 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 16/32 2023-10-11T10:39:00.232 INFO:teuthology.orchestra.run.smithi050.stdout:(14/32): ceph-mon-18.0.0-6661.g4bf94f59.el8.x86 2.2 MB/s | 4.6 MB 00:02 2023-10-11T10:39:00.273 INFO:teuthology.orchestra.run.smithi050.stdout:(15/32): python3-portend-2.6-1.el8.noarch.rpm 151 kB/s | 16 kB 00:00 2023-10-11T10:39:00.323 INFO:teuthology.orchestra.run.smithi050.stdout:(16/32): python3-simplegeneric-0.8.1-17.el8.noa 211 kB/s | 19 kB 00:00 2023-10-11T10:39:00.329 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 17/32 2023-10-11T10:39:00.366 INFO:teuthology.orchestra.run.smithi050.stdout:(17/32): python3-tempora-1.14.1-5.el8.noarch.rp 741 kB/s | 29 kB 00:00 2023-10-11T10:39:00.392 INFO:teuthology.orchestra.run.smithi050.stdout:(18/32): python3-pecan-1.3.2-9.el8.noarch.rpm 1.1 MB/s | 283 kB 00:00 2023-10-11T10:39:00.417 INFO:teuthology.orchestra.run.smithi050.stdout:(19/32): python3-trustme-0.6.0-4.el8.noarch.rpm 556 kB/s | 27 kB 00:00 2023-10-11T10:39:00.442 INFO:teuthology.orchestra.run.smithi050.stdout:(20/32): python3-waitress-1.4.3-1.el8.noarch.rp 5.0 MB/s | 255 kB 00:00 2023-10-11T10:39:00.468 INFO:teuthology.orchestra.run.smithi050.stdout:(21/32): python3-singledispatch-3.4.0.3-18.el8. 118 kB/s | 24 kB 00:00 2023-10-11T10:39:00.491 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 18/32 2023-10-11T10:39:00.493 INFO:teuthology.orchestra.run.smithi050.stdout:(22/32): python3-webtest-2.0.33-1.el8.noarch.rp 1.7 MB/s | 85 kB 00:00 2023-10-11T10:39:00.518 INFO:teuthology.orchestra.run.smithi050.stdout:(23/32): python3-zc-lockfile-2.0-2.el8.noarch.r 459 kB/s | 23 kB 00:00 2023-10-11T10:39:00.544 INFO:teuthology.orchestra.run.smithi050.stdout:(24/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 1.9 MB/s | 251 kB 00:00 2023-10-11T10:39:00.677 INFO:teuthology.orchestra.run.smithi050.stdout:(25/32): python3-pycparser-2.14-14.el8.noarch.r 594 kB/s | 109 kB 00:00 2023-10-11T10:39:00.713 INFO:teuthology.orchestra.run.smithi050.stdout:(26/32): python3-cryptography-3.2.1-5.el8.x86_6 2.8 MB/s | 559 kB 00:00 2023-10-11T10:39:00.736 INFO:teuthology.orchestra.run.smithi050.stdout:(27/32): python3-cffi-1.11.5-5.el8.x86_64.rpm 1.3 MB/s | 238 kB 00:00 2023-10-11T10:39:00.755 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 19/32 2023-10-11T10:39:00.820 INFO:teuthology.orchestra.run.smithi050.stdout:(28/32): python3-pytz-2017.2-9.el8.noarch.rpm 381 kB/s | 54 kB 00:00 2023-10-11T10:39:00.862 INFO:teuthology.orchestra.run.smithi050.stdout:(29/32): python3-markupsafe-0.23-19.el8.x86_64. 258 kB/s | 39 kB 00:00 2023-10-11T10:39:00.887 INFO:teuthology.orchestra.run.smithi050.stdout:(30/32): python3-mako-1.0.6-13.el8.noarch.rpm 1.0 MB/s | 157 kB 00:00 2023-10-11T10:39:00.954 INFO:teuthology.orchestra.run.smithi050.stdout:(31/32): python3-werkzeug-0.12.2-4.el8.noarch.r 3.3 MB/s | 457 kB 00:00 2023-10-11T10:39:00.963 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 20/32 2023-10-11T10:39:01.062 INFO:teuthology.orchestra.run.smithi050.stdout:(32/32): python3-pyOpenSSL-19.0.0-1.el8.noarch. 517 kB/s | 103 kB 00:00 2023-10-11T10:39:01.067 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2023-10-11T10:39:01.067 INFO:teuthology.orchestra.run.smithi050.stdout:Total 8.9 MB/s | 30 MB 00:03 2023-10-11T10:39:01.161 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2023-10-11T10:39:01.182 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2023-10-11T10:39:01.202 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2023-10-11T10:39:01.203 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2023-10-11T10:39:01.429 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2023-10-11T10:39:01.434 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2023-10-11T10:39:01.457 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2023-10-11T10:39:01.725 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2023-10-11T10:39:01.922 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 24/32 2023-10-11T10:39:02.546 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2023-10-11T10:39:02.764 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2023-10-11T10:39:03.101 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 25/32 2023-10-11T10:39:03.177 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2023-10-11T10:39:03.366 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 3/32 2023-10-11T10:39:03.451 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2023-10-11T10:39:03.626 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 4/32 2023-10-11T10:39:03.813 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-6661.g4bf94f59.el8. 27/32 2023-10-11T10:39:03.835 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 5/32 2023-10-11T10:39:04.003 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-mgr-2:18.0.0-6661.g4bf94f59.el8.x86_64 28/32 2023-10-11T10:39:04.018 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 6/32 2023-10-11T10:39:04.243 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 7/32 2023-10-11T10:39:04.675 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 8/32 2023-10-11T10:39:04.966 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-cryptography-3.2.1-5.el8.x86_64 9/32 2023-10-11T10:39:05.145 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 10/32 2023-10-11T10:39:05.321 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 11/32 2023-10-11T10:39:05.601 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-mgr-2:18.0.0-6661.g4bf94f59.el8.x86_64 28/32 2023-10-11T10:39:05.845 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-osd-2:18.0.0-6661.g4bf94f59.el8.x86_64 29/32 2023-10-11T10:39:06.263 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 12/32 2023-10-11T10:39:06.301 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-osd-2:18.0.0-6661.g4bf94f59.el8.x86_64 29/32 2023-10-11T10:39:06.450 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-mon-2:18.0.0-6661.g4bf94f59.el8.x86_64 30/32 2023-10-11T10:39:06.547 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 13/32 2023-10-11T10:39:06.666 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-mon-2:18.0.0-6661.g4bf94f59.el8.x86_64 30/32 2023-10-11T10:39:06.727 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 14/32 2023-10-11T10:39:06.875 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-mds-2:18.0.0-6661.g4bf94f59.el8.x86_64 31/32 2023-10-11T10:39:06.893 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-mds-2:18.0.0-6661.g4bf94f59.el8.x86_64 31/32 2023-10-11T10:39:06.908 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 15/32 2023-10-11T10:39:06.989 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-2:18.0.0-6661.g4bf94f59.el8.x86_64 32/32 2023-10-11T10:39:07.074 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 16/32 2023-10-11T10:39:07.248 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 17/32 2023-10-11T10:39:07.336 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-2:18.0.0-6661.g4bf94f59.el8.x86_64 32/32 2023-10-11T10:39:07.337 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-2:18.0.0-6661.g4bf94f59.el8.x86_64 1/32 2023-10-11T10:39:07.337 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-mds-2:18.0.0-6661.g4bf94f59.el8.x86_64 2/32 2023-10-11T10:39:07.337 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-mgr-2:18.0.0-6661.g4bf94f59.el8.x86_64 3/32 2023-10-11T10:39:07.337 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-mon-2:18.0.0-6661.g4bf94f59.el8.x86_64 4/32 2023-10-11T10:39:07.337 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-osd-2:18.0.0-6661.g4bf94f59.el8.x86_64 5/32 2023-10-11T10:39:07.337 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-6661.g4bf94f59.el8. 6/32 2023-10-11T10:39:07.338 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 7/32 2023-10-11T10:39:07.338 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 8/32 2023-10-11T10:39:07.338 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 9/32 2023-10-11T10:39:07.338 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 10/32 2023-10-11T10:39:07.338 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 11/32 2023-10-11T10:39:07.339 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 12/32 2023-10-11T10:39:07.339 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 13/32 2023-10-11T10:39:07.339 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 14/32 2023-10-11T10:39:07.339 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 15/32 2023-10-11T10:39:07.339 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-portend-2.6-1.el8.noarch 16/32 2023-10-11T10:39:07.340 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 17/32 2023-10-11T10:39:07.340 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 18/32 2023-10-11T10:39:07.340 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 19/32 2023-10-11T10:39:07.340 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 20/32 2023-10-11T10:39:07.340 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 21/32 2023-10-11T10:39:07.341 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 22/32 2023-10-11T10:39:07.341 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 23/32 2023-10-11T10:39:07.341 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 24/32 2023-10-11T10:39:07.341 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 25/32 2023-10-11T10:39:07.341 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-cryptography-3.2.1-5.el8.x86_64 26/32 2023-10-11T10:39:07.341 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 27/32 2023-10-11T10:39:07.342 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 28/32 2023-10-11T10:39:07.342 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 29/32 2023-10-11T10:39:07.342 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 30/32 2023-10-11T10:39:07.342 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 31/32 2023-10-11T10:39:07.407 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 18/32 2023-10-11T10:39:07.653 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 19/32 2023-10-11T10:39:07.866 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 20/32 2023-10-11T10:39:08.051 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2023-10-11T10:39:08.306 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2023-10-11T10:39:08.579 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2023-10-11T10:39:08.781 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 24/32 2023-10-11T10:39:08.917 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 32/32 2023-10-11T10:39:08.917 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:01:29 ago on Wed 11 Oct 2023 10:37:39 AM UTC. 2023-10-11T10:39:09.110 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 25/32 2023-10-11T10:39:09.449 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2023-10-11T10:39:09.789 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-6661.g4bf94f59.el8. 27/32 2023-10-11T10:39:09.936 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-mgr-2:18.0.0-6661.g4bf94f59.el8.x86_64 28/32 2023-10-11T10:39:11.647 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-mgr-2:18.0.0-6661.g4bf94f59.el8.x86_64 28/32 2023-10-11T10:39:11.836 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-osd-2:18.0.0-6661.g4bf94f59.el8.x86_64 29/32 2023-10-11T10:39:12.325 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-osd-2:18.0.0-6661.g4bf94f59.el8.x86_64 29/32 2023-10-11T10:39:12.469 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-mon-2:18.0.0-6661.g4bf94f59.el8.x86_64 30/32 2023-10-11T10:39:12.576 INFO:teuthology.orchestra.run.smithi111.stdout:Installed products updated. 2023-10-11T10:39:12.698 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-mon-2:18.0.0-6661.g4bf94f59.el8.x86_64 30/32 2023-10-11T10:39:12.760 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T10:39:12.761 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2023-10-11T10:39:12.761 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-2:18.0.0-6661.g4bf94f59.el8.x86_64 2023-10-11T10:39:12.761 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mds-2:18.0.0-6661.g4bf94f59.el8.x86_64 2023-10-11T10:39:12.761 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mgr-2:18.0.0-6661.g4bf94f59.el8.x86_64 2023-10-11T10:39:12.761 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mgr-modules-core-2:18.0.0-6661.g4bf94f59.el8.noarch 2023-10-11T10:39:12.762 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mon-2:18.0.0-6661.g4bf94f59.el8.x86_64 2023-10-11T10:39:12.762 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-osd-2:18.0.0-6661.g4bf94f59.el8.x86_64 2023-10-11T10:39:12.762 INFO:teuthology.orchestra.run.smithi111.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-10-11T10:39:12.762 INFO:teuthology.orchestra.run.smithi111.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-10-11T10:39:12.762 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cffi-1.11.5-5.el8.x86_64 2023-10-11T10:39:12.763 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-10-11T10:39:12.763 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-10-11T10:39:12.763 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cryptography-3.2.1-5.el8.x86_64 2023-10-11T10:39:12.763 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jaraco-6.2-6.el8.noarch 2023-10-11T10:39:12.763 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-10-11T10:39:12.764 INFO:teuthology.orchestra.run.smithi111.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-10-11T10:39:12.764 INFO:teuthology.orchestra.run.smithi111.stdout: python3-mako-1.0.6-13.el8.noarch 2023-10-11T10:39:12.764 INFO:teuthology.orchestra.run.smithi111.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-10-11T10:39:12.764 INFO:teuthology.orchestra.run.smithi111.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-10-11T10:39:12.764 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-10-11T10:39:12.764 INFO:teuthology.orchestra.run.smithi111.stdout: python3-portend-2.6-1.el8.noarch 2023-10-11T10:39:12.765 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-10-11T10:39:12.765 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pycparser-2.14-14.el8.noarch 2023-10-11T10:39:12.765 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pytz-2017.2-9.el8.noarch 2023-10-11T10:39:12.765 INFO:teuthology.orchestra.run.smithi111.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-10-11T10:39:12.765 INFO:teuthology.orchestra.run.smithi111.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-10-11T10:39:12.766 INFO:teuthology.orchestra.run.smithi111.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-10-11T10:39:12.766 INFO:teuthology.orchestra.run.smithi111.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-10-11T10:39:12.766 INFO:teuthology.orchestra.run.smithi111.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-10-11T10:39:12.766 INFO:teuthology.orchestra.run.smithi111.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-10-11T10:39:12.766 INFO:teuthology.orchestra.run.smithi111.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-10-11T10:39:12.766 INFO:teuthology.orchestra.run.smithi111.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-10-11T10:39:12.767 INFO:teuthology.orchestra.run.smithi111.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-10-11T10:39:12.767 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T10:39:12.767 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2023-10-11T10:39:12.869 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-mds-2:18.0.0-6661.g4bf94f59.el8.x86_64 31/32 2023-10-11T10:39:12.887 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-mds-2:18.0.0-6661.g4bf94f59.el8.x86_64 31/32 2023-10-11T10:39:12.919 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install ceph-base 2023-10-11T10:39:12.992 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-2:18.0.0-6661.g4bf94f59.el8.x86_64 32/32 2023-10-11T10:39:13.264 INFO:teuthology.orchestra.run.smithi111.stdout:Updating Subscription Management repositories. 2023-10-11T10:39:13.347 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-2:18.0.0-6661.g4bf94f59.el8.x86_64 32/32 2023-10-11T10:39:13.347 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-2:18.0.0-6661.g4bf94f59.el8.x86_64 1/32 2023-10-11T10:39:13.347 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-mds-2:18.0.0-6661.g4bf94f59.el8.x86_64 2/32 2023-10-11T10:39:13.348 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-mgr-2:18.0.0-6661.g4bf94f59.el8.x86_64 3/32 2023-10-11T10:39:13.348 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-mon-2:18.0.0-6661.g4bf94f59.el8.x86_64 4/32 2023-10-11T10:39:13.348 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-osd-2:18.0.0-6661.g4bf94f59.el8.x86_64 5/32 2023-10-11T10:39:13.348 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-6661.g4bf94f59.el8. 6/32 2023-10-11T10:39:13.348 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 7/32 2023-10-11T10:39:13.349 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 8/32 2023-10-11T10:39:13.349 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 9/32 2023-10-11T10:39:13.349 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 10/32 2023-10-11T10:39:13.349 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 11/32 2023-10-11T10:39:13.349 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 12/32 2023-10-11T10:39:13.350 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 13/32 2023-10-11T10:39:13.350 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 14/32 2023-10-11T10:39:13.350 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 15/32 2023-10-11T10:39:13.350 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-portend-2.6-1.el8.noarch 16/32 2023-10-11T10:39:13.350 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 17/32 2023-10-11T10:39:13.351 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 18/32 2023-10-11T10:39:13.351 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 19/32 2023-10-11T10:39:13.351 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 20/32 2023-10-11T10:39:13.351 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 21/32 2023-10-11T10:39:13.351 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 22/32 2023-10-11T10:39:13.352 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 23/32 2023-10-11T10:39:13.352 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 24/32 2023-10-11T10:39:13.352 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 25/32 2023-10-11T10:39:13.352 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-cryptography-3.2.1-5.el8.x86_64 26/32 2023-10-11T10:39:13.352 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 27/32 2023-10-11T10:39:13.353 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 28/32 2023-10-11T10:39:13.353 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 29/32 2023-10-11T10:39:13.353 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 30/32 2023-10-11T10:39:13.353 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 31/32 2023-10-11T10:39:14.146 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:01:35 ago on Wed 11 Oct 2023 10:37:39 AM UTC. 2023-10-11T10:39:14.947 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 32/32 2023-10-11T10:39:14.948 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:01:33 ago on Wed 11 Oct 2023 10:37:41 AM UTC. 2023-10-11T10:39:15.434 INFO:teuthology.orchestra.run.smithi111.stdout:Package ceph-base-2:18.0.0-6661.g4bf94f59.el8.x86_64 is already installed. 2023-10-11T10:39:15.543 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2023-10-11T10:39:15.544 INFO:teuthology.orchestra.run.smithi111.stdout:Nothing to do. 2023-10-11T10:39:15.544 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2023-10-11T10:39:15.663 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install cephadm 2023-10-11T10:39:16.003 INFO:teuthology.orchestra.run.smithi111.stdout:Updating Subscription Management repositories. 2023-10-11T10:39:16.844 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:01:37 ago on Wed 11 Oct 2023 10:37:39 AM UTC. 2023-10-11T10:39:17.731 INFO:teuthology.orchestra.run.smithi050.stdout:Installed products updated. 2023-10-11T10:39:17.934 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-11T10:39:17.935 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2023-10-11T10:39:17.935 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-2:18.0.0-6661.g4bf94f59.el8.x86_64 2023-10-11T10:39:17.935 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mds-2:18.0.0-6661.g4bf94f59.el8.x86_64 2023-10-11T10:39:17.935 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-2:18.0.0-6661.g4bf94f59.el8.x86_64 2023-10-11T10:39:17.936 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-modules-core-2:18.0.0-6661.g4bf94f59.el8.noarch 2023-10-11T10:39:17.936 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mon-2:18.0.0-6661.g4bf94f59.el8.x86_64 2023-10-11T10:39:17.936 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-osd-2:18.0.0-6661.g4bf94f59.el8.x86_64 2023-10-11T10:39:17.936 INFO:teuthology.orchestra.run.smithi050.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-10-11T10:39:17.936 INFO:teuthology.orchestra.run.smithi050.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-10-11T10:39:17.937 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cffi-1.11.5-5.el8.x86_64 2023-10-11T10:39:17.937 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-10-11T10:39:17.937 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-10-11T10:39:17.937 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cryptography-3.2.1-5.el8.x86_64 2023-10-11T10:39:17.937 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jaraco-6.2-6.el8.noarch 2023-10-11T10:39:17.938 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-10-11T10:39:17.938 INFO:teuthology.orchestra.run.smithi050.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-10-11T10:39:17.938 INFO:teuthology.orchestra.run.smithi050.stdout: python3-mako-1.0.6-13.el8.noarch 2023-10-11T10:39:17.938 INFO:teuthology.orchestra.run.smithi050.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-10-11T10:39:17.938 INFO:teuthology.orchestra.run.smithi050.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-10-11T10:39:17.939 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-10-11T10:39:17.939 INFO:teuthology.orchestra.run.smithi050.stdout: python3-portend-2.6-1.el8.noarch 2023-10-11T10:39:17.939 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-10-11T10:39:17.939 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pycparser-2.14-14.el8.noarch 2023-10-11T10:39:17.939 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pytz-2017.2-9.el8.noarch 2023-10-11T10:39:17.939 INFO:teuthology.orchestra.run.smithi050.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-10-11T10:39:17.940 INFO:teuthology.orchestra.run.smithi050.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-10-11T10:39:17.940 INFO:teuthology.orchestra.run.smithi050.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-10-11T10:39:17.940 INFO:teuthology.orchestra.run.smithi050.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-10-11T10:39:17.940 INFO:teuthology.orchestra.run.smithi050.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-10-11T10:39:17.941 INFO:teuthology.orchestra.run.smithi050.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-10-11T10:39:17.941 INFO:teuthology.orchestra.run.smithi050.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-10-11T10:39:17.941 INFO:teuthology.orchestra.run.smithi050.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-10-11T10:39:17.941 INFO:teuthology.orchestra.run.smithi050.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-10-11T10:39:17.942 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-11T10:39:17.942 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2023-10-11T10:39:18.126 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install ceph-base 2023-10-11T10:39:18.210 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2023-10-11T10:39:18.211 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2023-10-11T10:39:18.211 INFO:teuthology.orchestra.run.smithi111.stdout: Package Arch Version Repository Size 2023-10-11T10:39:18.212 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2023-10-11T10:39:18.212 INFO:teuthology.orchestra.run.smithi111.stdout:Installing: 2023-10-11T10:39:18.212 INFO:teuthology.orchestra.run.smithi111.stdout: cephadm noarch 2:18.0.0-6661.g4bf94f59.el8 ceph-noarch 190 k 2023-10-11T10:39:18.212 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T10:39:18.213 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2023-10-11T10:39:18.213 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2023-10-11T10:39:18.213 INFO:teuthology.orchestra.run.smithi111.stdout:Install 1 Package 2023-10-11T10:39:18.213 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T10:39:18.213 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 190 k 2023-10-11T10:39:18.213 INFO:teuthology.orchestra.run.smithi111.stdout:Installed size: 776 k 2023-10-11T10:39:18.214 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2023-10-11T10:39:18.496 INFO:teuthology.orchestra.run.smithi050.stdout:Updating Subscription Management repositories. 2023-10-11T10:39:18.546 INFO:teuthology.orchestra.run.smithi111.stdout:cephadm-18.0.0-6661.g4bf94f59.el8.noarch.rpm 569 kB/s | 190 kB 00:00 2023-10-11T10:39:18.547 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2023-10-11T10:39:18.547 INFO:teuthology.orchestra.run.smithi111.stdout:Total 567 kB/s | 190 kB 00:00 2023-10-11T10:39:18.547 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2023-10-11T10:39:18.555 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2023-10-11T10:39:18.555 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2023-10-11T10:39:18.557 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2023-10-11T10:39:18.561 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2023-10-11T10:39:18.603 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2023-10-11T10:39:19.370 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:01:38 ago on Wed 11 Oct 2023 10:37:41 AM UTC. 2023-10-11T10:39:20.335 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: cephadm-2:18.0.0-6661.g4bf94f59.el8.noarch 1/1 2023-10-11T10:39:20.479 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : cephadm-2:18.0.0-6661.g4bf94f59.el8.noarch 1/1 2023-10-11T10:39:20.646 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: cephadm-2:18.0.0-6661.g4bf94f59.el8.noarch 1/1 2023-10-11T10:39:20.808 INFO:teuthology.orchestra.run.smithi050.stdout:Package ceph-base-2:18.0.0-6661.g4bf94f59.el8.x86_64 is already installed. 2023-10-11T10:39:20.926 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2023-10-11T10:39:20.928 INFO:teuthology.orchestra.run.smithi050.stdout:Nothing to do. 2023-10-11T10:39:20.928 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2023-10-11T10:39:21.061 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install cephadm 2023-10-11T10:39:21.425 INFO:teuthology.orchestra.run.smithi050.stdout:Updating Subscription Management repositories. 2023-10-11T10:39:21.692 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : cephadm-2:18.0.0-6661.g4bf94f59.el8.noarch 1/1 2023-10-11T10:39:21.693 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:01:42 ago on Wed 11 Oct 2023 10:37:39 AM UTC. 2023-10-11T10:39:23.001 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:01:42 ago on Wed 11 Oct 2023 10:37:41 AM UTC. 2023-10-11T10:39:24.197 INFO:teuthology.orchestra.run.smithi111.stdout:Installed products updated. 2023-10-11T10:39:24.374 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T10:39:24.374 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2023-10-11T10:39:24.375 INFO:teuthology.orchestra.run.smithi111.stdout: cephadm-2:18.0.0-6661.g4bf94f59.el8.noarch 2023-10-11T10:39:24.375 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T10:39:24.375 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2023-10-11T10:39:24.501 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2023-10-11T10:39:24.503 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-10-11T10:39:24.503 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repository Size 2023-10-11T10:39:24.503 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-10-11T10:39:24.503 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2023-10-11T10:39:24.503 INFO:teuthology.orchestra.run.smithi050.stdout: cephadm noarch 2:18.0.0-6661.g4bf94f59.el8 ceph-noarch 190 k 2023-10-11T10:39:24.504 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-11T10:39:24.504 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2023-10-11T10:39:24.504 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-10-11T10:39:24.504 INFO:teuthology.orchestra.run.smithi050.stdout:Install 1 Package 2023-10-11T10:39:24.504 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-11T10:39:24.505 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 190 k 2023-10-11T10:39:24.505 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 776 k 2023-10-11T10:39:24.505 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2023-10-11T10:39:24.523 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install ceph-immutable-object-cache 2023-10-11T10:39:24.785 INFO:teuthology.orchestra.run.smithi050.stdout:cephadm-18.0.0-6661.g4bf94f59.el8.noarch.rpm 674 kB/s | 190 kB 00:00 2023-10-11T10:39:24.786 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2023-10-11T10:39:24.786 INFO:teuthology.orchestra.run.smithi050.stdout:Total 671 kB/s | 190 kB 00:00 2023-10-11T10:39:24.786 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2023-10-11T10:39:24.795 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2023-10-11T10:39:24.795 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2023-10-11T10:39:24.797 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2023-10-11T10:39:24.801 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2023-10-11T10:39:24.842 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2023-10-11T10:39:24.879 INFO:teuthology.orchestra.run.smithi111.stdout:Updating Subscription Management repositories. 2023-10-11T10:39:25.659 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:01:46 ago on Wed 11 Oct 2023 10:37:39 AM UTC. 2023-10-11T10:39:26.610 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: cephadm-2:18.0.0-6661.g4bf94f59.el8.noarch 1/1 2023-10-11T10:39:26.771 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : cephadm-2:18.0.0-6661.g4bf94f59.el8.noarch 1/1 2023-10-11T10:39:26.945 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: cephadm-2:18.0.0-6661.g4bf94f59.el8.noarch 1/1 2023-10-11T10:39:27.033 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2023-10-11T10:39:27.034 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2023-10-11T10:39:27.034 INFO:teuthology.orchestra.run.smithi111.stdout: Package Arch Version Repo Size 2023-10-11T10:39:27.034 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2023-10-11T10:39:27.034 INFO:teuthology.orchestra.run.smithi111.stdout:Installing: 2023-10-11T10:39:27.035 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-6661.g4bf94f59.el8 ceph 167 k 2023-10-11T10:39:27.035 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T10:39:27.035 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2023-10-11T10:39:27.035 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2023-10-11T10:39:27.035 INFO:teuthology.orchestra.run.smithi111.stdout:Install 1 Package 2023-10-11T10:39:27.036 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T10:39:27.036 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 167 k 2023-10-11T10:39:27.036 INFO:teuthology.orchestra.run.smithi111.stdout:Installed size: 512 k 2023-10-11T10:39:27.037 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2023-10-11T10:39:27.368 INFO:teuthology.orchestra.run.smithi111.stdout:ceph-immutable-object-cache-18.0.0-6661.g4bf94f 501 kB/s | 167 kB 00:00 2023-10-11T10:39:27.369 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2023-10-11T10:39:27.369 INFO:teuthology.orchestra.run.smithi111.stdout:Total 499 kB/s | 167 kB 00:00 2023-10-11T10:39:27.370 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2023-10-11T10:39:27.380 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2023-10-11T10:39:27.381 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2023-10-11T10:39:27.431 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2023-10-11T10:39:27.435 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2023-10-11T10:39:27.593 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2023-10-11T10:39:27.730 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-6661.g4bf94f59. 1/1 2023-10-11T10:39:28.014 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : cephadm-2:18.0.0-6661.g4bf94f59.el8.noarch 1/1 2023-10-11T10:39:28.014 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:01:47 ago on Wed 11 Oct 2023 10:37:41 AM UTC. 2023-10-11T10:39:28.108 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-6661.g4bf94f59. 1/1 2023-10-11T10:39:29.114 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-6661.g4bf94f59. 1/1 2023-10-11T10:39:29.115 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:01:50 ago on Wed 11 Oct 2023 10:37:39 AM UTC. 2023-10-11T10:39:30.615 INFO:teuthology.orchestra.run.smithi050.stdout:Installed products updated. 2023-10-11T10:39:30.813 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-11T10:39:30.814 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2023-10-11T10:39:30.814 INFO:teuthology.orchestra.run.smithi050.stdout: cephadm-2:18.0.0-6661.g4bf94f59.el8.noarch 2023-10-11T10:39:30.814 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-11T10:39:30.814 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2023-10-11T10:39:30.962 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install ceph-immutable-object-cache 2023-10-11T10:39:31.343 INFO:teuthology.orchestra.run.smithi050.stdout:Updating Subscription Management repositories. 2023-10-11T10:39:31.608 INFO:teuthology.orchestra.run.smithi111.stdout:Installed products updated. 2023-10-11T10:39:31.786 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T10:39:31.787 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2023-10-11T10:39:31.787 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-immutable-object-cache-2:18.0.0-6661.g4bf94f59.el8.x86_64 2023-10-11T10:39:31.787 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T10:39:31.787 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2023-10-11T10:39:31.916 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install ceph-mgr 2023-10-11T10:39:32.198 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:01:51 ago on Wed 11 Oct 2023 10:37:41 AM UTC. 2023-10-11T10:39:32.260 INFO:teuthology.orchestra.run.smithi111.stdout:Updating Subscription Management repositories. 2023-10-11T10:39:33.084 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:01:54 ago on Wed 11 Oct 2023 10:37:39 AM UTC. 2023-10-11T10:39:33.740 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2023-10-11T10:39:33.741 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-10-11T10:39:33.741 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repo Size 2023-10-11T10:39:33.742 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-10-11T10:39:33.742 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2023-10-11T10:39:33.742 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-6661.g4bf94f59.el8 ceph 167 k 2023-10-11T10:39:33.742 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-11T10:39:33.742 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2023-10-11T10:39:33.743 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-10-11T10:39:33.743 INFO:teuthology.orchestra.run.smithi050.stdout:Install 1 Package 2023-10-11T10:39:33.743 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-11T10:39:33.743 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 167 k 2023-10-11T10:39:33.743 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 512 k 2023-10-11T10:39:33.743 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2023-10-11T10:39:33.932 INFO:teuthology.orchestra.run.smithi050.stdout:ceph-immutable-object-cache-18.0.0-6661.g4bf94f 879 kB/s | 167 kB 00:00 2023-10-11T10:39:33.933 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2023-10-11T10:39:33.933 INFO:teuthology.orchestra.run.smithi050.stdout:Total 873 kB/s | 167 kB 00:00 2023-10-11T10:39:33.933 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2023-10-11T10:39:33.946 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2023-10-11T10:39:33.946 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2023-10-11T10:39:34.001 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2023-10-11T10:39:34.005 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2023-10-11T10:39:34.128 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2023-10-11T10:39:34.266 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-6661.g4bf94f59. 1/1 2023-10-11T10:39:34.429 INFO:teuthology.orchestra.run.smithi111.stdout:Package ceph-mgr-2:18.0.0-6661.g4bf94f59.el8.x86_64 is already installed. 2023-10-11T10:39:34.537 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2023-10-11T10:39:34.538 INFO:teuthology.orchestra.run.smithi111.stdout:Nothing to do. 2023-10-11T10:39:34.538 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2023-10-11T10:39:34.650 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install ceph-mgr-dashboard 2023-10-11T10:39:34.670 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-6661.g4bf94f59. 1/1 2023-10-11T10:39:34.994 INFO:teuthology.orchestra.run.smithi111.stdout:Updating Subscription Management repositories. 2023-10-11T10:39:35.756 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-6661.g4bf94f59. 1/1 2023-10-11T10:39:35.756 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:01:54 ago on Wed 11 Oct 2023 10:37:41 AM UTC. 2023-10-11T10:39:35.869 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:01:56 ago on Wed 11 Oct 2023 10:37:39 AM UTC. 2023-10-11T10:39:37.333 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2023-10-11T10:39:37.334 INFO:teuthology.orchestra.run.smithi111.stdout:=================================================================================================== 2023-10-11T10:39:37.334 INFO:teuthology.orchestra.run.smithi111.stdout: Package Arch Version Repository Size 2023-10-11T10:39:37.334 INFO:teuthology.orchestra.run.smithi111.stdout:=================================================================================================== 2023-10-11T10:39:37.334 INFO:teuthology.orchestra.run.smithi111.stdout:Installing: 2023-10-11T10:39:37.335 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mgr-dashboard noarch 2:18.0.0-6661.g4bf94f59.el8 ceph-noarch 1.8 M 2023-10-11T10:39:37.335 INFO:teuthology.orchestra.run.smithi111.stdout:Installing dependencies: 2023-10-11T10:39:37.335 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-grafana-dashboards noarch 2:18.0.0-6661.g4bf94f59.el8 ceph-noarch 23 k 2023-10-11T10:39:37.335 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-prometheus-alerts noarch 2:18.0.0-6661.g4bf94f59.el8 ceph-noarch 15 k 2023-10-11T10:39:37.335 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-10-11T10:39:37.336 INFO:teuthology.orchestra.run.smithi111.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-10-11T10:39:37.336 INFO:teuthology.orchestra.run.smithi111.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-10-11T10:39:37.336 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T10:39:37.336 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2023-10-11T10:39:37.336 INFO:teuthology.orchestra.run.smithi111.stdout:=================================================================================================== 2023-10-11T10:39:37.337 INFO:teuthology.orchestra.run.smithi111.stdout:Install 6 Packages 2023-10-11T10:39:37.337 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T10:39:37.337 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 2.1 M 2023-10-11T10:39:37.338 INFO:teuthology.orchestra.run.smithi111.stdout:Installed size: 8.5 M 2023-10-11T10:39:37.338 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2023-10-11T10:39:37.808 INFO:teuthology.orchestra.run.smithi111.stdout:(1/6): ceph-grafana-dashboards-18.0.0-6661.g4bf 79 kB/s | 23 kB 00:00 2023-10-11T10:39:37.833 INFO:teuthology.orchestra.run.smithi111.stdout:(2/6): ceph-prometheus-alerts-18.0.0-6661.g4bf9 46 kB/s | 15 kB 00:00 2023-10-11T10:39:37.875 INFO:teuthology.orchestra.run.smithi111.stdout:(3/6): ceph-mgr-dashboard-18.0.0-6661.g4bf94f59 4.9 MB/s | 1.8 MB 00:00 2023-10-11T10:39:37.942 INFO:teuthology.orchestra.run.smithi111.stdout:(4/6): python3-jwt-1.6.1-2.el8.noarch.rpm 641 kB/s | 43 kB 00:00 2023-10-11T10:39:37.967 INFO:teuthology.orchestra.run.smithi111.stdout:(5/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 211 kB/s | 34 kB 00:00 2023-10-11T10:39:38.034 INFO:teuthology.orchestra.run.smithi111.stdout:(6/6): python3-routes-2.4.1-12.el8.noarch.rpm 978 kB/s | 196 kB 00:00 2023-10-11T10:39:38.036 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2023-10-11T10:39:38.036 INFO:teuthology.orchestra.run.smithi111.stdout:Total 2.9 MB/s | 2.1 MB 00:00 2023-10-11T10:39:38.047 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2023-10-11T10:39:38.056 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2023-10-11T10:39:38.057 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2023-10-11T10:39:38.112 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2023-10-11T10:39:38.116 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2023-10-11T10:39:38.379 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2023-10-11T10:39:38.546 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 1/6 2023-10-11T10:39:38.577 INFO:teuthology.orchestra.run.smithi050.stdout:Installed products updated. 2023-10-11T10:39:38.759 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 2/6 2023-10-11T10:39:38.778 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-11T10:39:38.778 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2023-10-11T10:39:38.779 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-immutable-object-cache-2:18.0.0-6661.g4bf94f59.el8.x86_64 2023-10-11T10:39:38.779 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-11T10:39:38.779 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2023-10-11T10:39:38.964 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 3/6 2023-10-11T10:39:39.109 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-6661.g4bf94f59.el8.n 4/6 2023-10-11T10:39:39.154 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install ceph-mgr 2023-10-11T10:39:39.488 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-6661.g4bf94f59.el8. 5/6 2023-10-11T10:39:39.521 INFO:teuthology.orchestra.run.smithi050.stdout:Updating Subscription Management repositories. 2023-10-11T10:39:39.746 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-6661.g4bf94f59.el8.noarc 6/6 2023-10-11T10:39:40.004 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-6661.g4bf94f59.el8.noarc 6/6 2023-10-11T10:39:40.005 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-6661.g4bf94f59.el8. 1/6 2023-10-11T10:39:40.005 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-6661.g4bf94f59.el8.noarc 2/6 2023-10-11T10:39:40.005 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-6661.g4bf94f59.el8.n 3/6 2023-10-11T10:39:40.005 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 4/6 2023-10-11T10:39:40.006 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 5/6 2023-10-11T10:39:41.009 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 6/6 2023-10-11T10:39:41.010 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:02:02 ago on Wed 11 Oct 2023 10:37:39 AM UTC. 2023-10-11T10:39:41.074 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:02:00 ago on Wed 11 Oct 2023 10:37:41 AM UTC. 2023-10-11T10:39:42.515 INFO:teuthology.orchestra.run.smithi050.stdout:Package ceph-mgr-2:18.0.0-6661.g4bf94f59.el8.x86_64 is already installed. 2023-10-11T10:39:42.639 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2023-10-11T10:39:42.641 INFO:teuthology.orchestra.run.smithi050.stdout:Nothing to do. 2023-10-11T10:39:42.641 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2023-10-11T10:39:42.769 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install ceph-mgr-dashboard 2023-10-11T10:39:43.134 INFO:teuthology.orchestra.run.smithi050.stdout:Updating Subscription Management repositories. 2023-10-11T10:39:44.745 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:02:03 ago on Wed 11 Oct 2023 10:37:41 AM UTC. 2023-10-11T10:39:45.878 INFO:teuthology.orchestra.run.smithi111.stdout:Installed products updated. 2023-10-11T10:39:46.059 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T10:39:46.059 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2023-10-11T10:39:46.059 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-grafana-dashboards-2:18.0.0-6661.g4bf94f59.el8.noarch 2023-10-11T10:39:46.059 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mgr-dashboard-2:18.0.0-6661.g4bf94f59.el8.noarch 2023-10-11T10:39:46.060 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-prometheus-alerts-2:18.0.0-6661.g4bf94f59.el8.noarch 2023-10-11T10:39:46.060 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-10-11T10:39:46.060 INFO:teuthology.orchestra.run.smithi111.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-10-11T10:39:46.060 INFO:teuthology.orchestra.run.smithi111.stdout: python3-routes-2.4.1-12.el8.noarch 2023-10-11T10:39:46.060 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T10:39:46.061 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2023-10-11T10:39:46.242 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install ceph-mgr-diskprediction-local 2023-10-11T10:39:46.297 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2023-10-11T10:39:46.298 INFO:teuthology.orchestra.run.smithi050.stdout:=================================================================================================== 2023-10-11T10:39:46.299 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repository Size 2023-10-11T10:39:46.299 INFO:teuthology.orchestra.run.smithi050.stdout:=================================================================================================== 2023-10-11T10:39:46.299 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2023-10-11T10:39:46.299 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-dashboard noarch 2:18.0.0-6661.g4bf94f59.el8 ceph-noarch 1.8 M 2023-10-11T10:39:46.299 INFO:teuthology.orchestra.run.smithi050.stdout:Installing dependencies: 2023-10-11T10:39:46.300 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-grafana-dashboards noarch 2:18.0.0-6661.g4bf94f59.el8 ceph-noarch 23 k 2023-10-11T10:39:46.300 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-prometheus-alerts noarch 2:18.0.0-6661.g4bf94f59.el8 ceph-noarch 15 k 2023-10-11T10:39:46.300 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-10-11T10:39:46.300 INFO:teuthology.orchestra.run.smithi050.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-10-11T10:39:46.300 INFO:teuthology.orchestra.run.smithi050.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-10-11T10:39:46.300 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-11T10:39:46.301 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2023-10-11T10:39:46.301 INFO:teuthology.orchestra.run.smithi050.stdout:=================================================================================================== 2023-10-11T10:39:46.301 INFO:teuthology.orchestra.run.smithi050.stdout:Install 6 Packages 2023-10-11T10:39:46.301 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-11T10:39:46.302 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 2.1 M 2023-10-11T10:39:46.302 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 8.5 M 2023-10-11T10:39:46.302 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2023-10-11T10:39:46.512 INFO:teuthology.orchestra.run.smithi050.stdout:(1/6): ceph-prometheus-alerts-18.0.0-6661.g4bf9 83 kB/s | 15 kB 00:00 2023-10-11T10:39:46.587 INFO:teuthology.orchestra.run.smithi111.stdout:Updating Subscription Management repositories. 2023-10-11T10:39:46.671 INFO:teuthology.orchestra.run.smithi050.stdout:(2/6): ceph-mgr-dashboard-18.0.0-6661.g4bf94f59 5.3 MB/s | 1.8 MB 00:00 2023-10-11T10:39:46.696 INFO:teuthology.orchestra.run.smithi050.stdout:(3/6): ceph-grafana-dashboards-18.0.0-6661.g4bf 64 kB/s | 23 kB 00:00 2023-10-11T10:39:46.763 INFO:teuthology.orchestra.run.smithi050.stdout:(4/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 134 kB/s | 34 kB 00:00 2023-10-11T10:39:46.897 INFO:teuthology.orchestra.run.smithi050.stdout:(5/6): python3-jwt-1.6.1-2.el8.noarch.rpm 212 kB/s | 43 kB 00:00 2023-10-11T10:39:47.022 INFO:teuthology.orchestra.run.smithi050.stdout:(6/6): python3-routes-2.4.1-12.el8.noarch.rpm 560 kB/s | 196 kB 00:00 2023-10-11T10:39:47.025 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2023-10-11T10:39:47.026 INFO:teuthology.orchestra.run.smithi050.stdout:Total 2.8 MB/s | 2.1 MB 00:00 2023-10-11T10:39:47.038 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2023-10-11T10:39:47.049 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2023-10-11T10:39:47.049 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2023-10-11T10:39:47.113 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2023-10-11T10:39:47.117 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2023-10-11T10:39:47.371 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2023-10-11T10:39:47.541 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 1/6 2023-10-11T10:39:47.732 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 2/6 2023-10-11T10:39:47.925 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 3/6 2023-10-11T10:39:48.008 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:02:09 ago on Wed 11 Oct 2023 10:37:39 AM UTC. 2023-10-11T10:39:48.062 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-6661.g4bf94f59.el8.n 4/6 2023-10-11T10:39:48.424 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-6661.g4bf94f59.el8. 5/6 2023-10-11T10:39:48.634 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-6661.g4bf94f59.el8.noarc 6/6 2023-10-11T10:39:48.880 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-6661.g4bf94f59.el8.noarc 6/6 2023-10-11T10:39:48.881 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-6661.g4bf94f59.el8. 1/6 2023-10-11T10:39:48.881 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-6661.g4bf94f59.el8.noarc 2/6 2023-10-11T10:39:48.881 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-6661.g4bf94f59.el8.n 3/6 2023-10-11T10:39:48.881 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 4/6 2023-10-11T10:39:48.882 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 5/6 2023-10-11T10:39:49.397 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2023-10-11T10:39:49.399 INFO:teuthology.orchestra.run.smithi111.stdout:======================================================================================================================= 2023-10-11T10:39:49.399 INFO:teuthology.orchestra.run.smithi111.stdout: Package Arch Version Repository Size 2023-10-11T10:39:49.399 INFO:teuthology.orchestra.run.smithi111.stdout:======================================================================================================================= 2023-10-11T10:39:49.399 INFO:teuthology.orchestra.run.smithi111.stdout:Installing: 2023-10-11T10:39:49.399 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-6661.g4bf94f59.el8 ceph-noarch 7.4 M 2023-10-11T10:39:49.400 INFO:teuthology.orchestra.run.smithi111.stdout:Installing dependencies: 2023-10-11T10:39:49.400 INFO:teuthology.orchestra.run.smithi111.stdout: libgfortran x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 644 k 2023-10-11T10:39:49.400 INFO:teuthology.orchestra.run.smithi111.stdout: libquadmath x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 171 k 2023-10-11T10:39:49.400 INFO:teuthology.orchestra.run.smithi111.stdout: openblas x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.7 M 2023-10-11T10:39:49.400 INFO:teuthology.orchestra.run.smithi111.stdout: openblas-threads x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.9 M 2023-10-11T10:39:49.401 INFO:teuthology.orchestra.run.smithi111.stdout: python3-numpy x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2023-10-11T10:39:49.401 INFO:teuthology.orchestra.run.smithi111.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2023-10-11T10:39:49.401 INFO:teuthology.orchestra.run.smithi111.stdout: python3-scipy x86_64 1.0.0-21.module+el8.5.0+10916+41bd434d rhel-8-for-x86_64-appstream-rpms 14 M 2023-10-11T10:39:49.401 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T10:39:49.401 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2023-10-11T10:39:49.401 INFO:teuthology.orchestra.run.smithi111.stdout:======================================================================================================================= 2023-10-11T10:39:49.402 INFO:teuthology.orchestra.run.smithi111.stdout:Install 8 Packages 2023-10-11T10:39:49.402 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T10:39:49.402 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 35 M 2023-10-11T10:39:49.403 INFO:teuthology.orchestra.run.smithi111.stdout:Installed size: 210 M 2023-10-11T10:39:49.403 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2023-10-11T10:39:49.590 INFO:teuthology.orchestra.run.smithi111.stdout:(1/8): libgfortran-8.5.0-10.1.el8_6.x86_64.rpm 3.3 MB/s | 644 kB 00:00 2023-10-11T10:39:49.615 INFO:teuthology.orchestra.run.smithi111.stdout:(2/8): libquadmath-8.5.0-10.1.el8_6.x86_64.rpm 801 kB/s | 171 kB 00:00 2023-10-11T10:39:49.799 INFO:teuthology.orchestra.run.smithi111.stdout:(3/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 20 MB/s | 3.7 MB 00:00 2023-10-11T10:39:49.957 INFO:teuthology.orchestra.run.smithi111.stdout:(4/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 1.4 MB/s | 225 kB 00:00 2023-10-11T10:39:49.971 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 6/6 2023-10-11T10:39:49.971 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:02:08 ago on Wed 11 Oct 2023 10:37:41 AM UTC. 2023-10-11T10:39:50.224 INFO:teuthology.orchestra.run.smithi111.stdout:(5/8): python3-scipy-1.0.0-21.module+el8.5.0+10 22 MB/s | 14 MB 00:00 2023-10-11T10:39:50.350 INFO:teuthology.orchestra.run.smithi111.stdout:(6/8): openblas-threads-0.3.15-3.el8.x86_64.rpm 12 MB/s | 4.9 MB 00:00 2023-10-11T10:39:50.442 INFO:teuthology.orchestra.run.smithi111.stdout:(7/8): ceph-mgr-diskprediction-local-18.0.0-666 7.1 MB/s | 7.4 MB 00:01 2023-10-11T10:39:50.525 INFO:teuthology.orchestra.run.smithi111.stdout:(8/8): openblas-0.3.15-3.el8.x86_64.rpm 16 MB/s | 4.7 MB 00:00 2023-10-11T10:39:50.526 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2023-10-11T10:39:50.526 INFO:teuthology.orchestra.run.smithi111.stdout:Total 31 MB/s | 35 MB 00:01 2023-10-11T10:39:50.762 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2023-10-11T10:39:50.780 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2023-10-11T10:39:50.781 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2023-10-11T10:39:51.071 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2023-10-11T10:39:51.075 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2023-10-11T10:39:51.641 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2023-10-11T10:39:51.787 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : libquadmath-8.5.0-10.1.el8_6.x86_64 1/8 2023-10-11T10:39:51.944 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: libquadmath-8.5.0-10.1.el8_6.x86_64 1/8 2023-10-11T10:39:52.497 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : libgfortran-8.5.0-10.1.el8_6.x86_64 2/8 2023-10-11T10:39:53.160 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: libgfortran-8.5.0-10.1.el8_6.x86_64 2/8 2023-10-11T10:39:53.376 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : openblas-threads-0.3.15-3.el8.x86_64 3/8 2023-10-11T10:39:53.957 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: openblas-threads-0.3.15-3.el8.x86_64 3/8 2023-10-11T10:39:54.125 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : openblas-0.3.15-3.el8.x86_64 4/8 2023-10-11T10:39:54.802 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: openblas-0.3.15-3.el8.x86_64 4/8 2023-10-11T10:39:54.996 INFO:teuthology.orchestra.run.smithi050.stdout:Installed products updated. 2023-10-11T10:39:55.102 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2023-10-11T10:39:55.193 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-11T10:39:55.194 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2023-10-11T10:39:55.194 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-grafana-dashboards-2:18.0.0-6661.g4bf94f59.el8.noarch 2023-10-11T10:39:55.194 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-dashboard-2:18.0.0-6661.g4bf94f59.el8.noarch 2023-10-11T10:39:55.194 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-prometheus-alerts-2:18.0.0-6661.g4bf94f59.el8.noarch 2023-10-11T10:39:55.194 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-10-11T10:39:55.195 INFO:teuthology.orchestra.run.smithi050.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-10-11T10:39:55.195 INFO:teuthology.orchestra.run.smithi050.stdout: python3-routes-2.4.1-12.el8.noarch 2023-10-11T10:39:55.195 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-11T10:39:55.195 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2023-10-11T10:39:55.284 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-10-11T10:39:55.388 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install ceph-mgr-diskprediction-local 2023-10-11T10:39:55.757 INFO:teuthology.orchestra.run.smithi050.stdout:Updating Subscription Management repositories. 2023-10-11T10:39:57.031 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-10-11T10:39:57.091 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:02:16 ago on Wed 11 Oct 2023 10:37:41 AM UTC. 2023-10-11T10:39:58.270 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd434d 7/8 2023-10-11T10:39:58.449 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-6661.g4bf94f5 8/8 2023-10-11T10:39:58.599 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2023-10-11T10:39:58.600 INFO:teuthology.orchestra.run.smithi050.stdout:======================================================================================================================= 2023-10-11T10:39:58.601 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repository Size 2023-10-11T10:39:58.601 INFO:teuthology.orchestra.run.smithi050.stdout:======================================================================================================================= 2023-10-11T10:39:58.601 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2023-10-11T10:39:58.601 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-6661.g4bf94f59.el8 ceph-noarch 7.4 M 2023-10-11T10:39:58.601 INFO:teuthology.orchestra.run.smithi050.stdout:Installing dependencies: 2023-10-11T10:39:58.602 INFO:teuthology.orchestra.run.smithi050.stdout: libgfortran x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 644 k 2023-10-11T10:39:58.602 INFO:teuthology.orchestra.run.smithi050.stdout: libquadmath x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 171 k 2023-10-11T10:39:58.602 INFO:teuthology.orchestra.run.smithi050.stdout: openblas x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.7 M 2023-10-11T10:39:58.602 INFO:teuthology.orchestra.run.smithi050.stdout: openblas-threads x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.9 M 2023-10-11T10:39:58.602 INFO:teuthology.orchestra.run.smithi050.stdout: python3-numpy x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2023-10-11T10:39:58.603 INFO:teuthology.orchestra.run.smithi050.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2023-10-11T10:39:58.603 INFO:teuthology.orchestra.run.smithi050.stdout: python3-scipy x86_64 1.0.0-21.module+el8.5.0+10916+41bd434d rhel-8-for-x86_64-appstream-rpms 14 M 2023-10-11T10:39:58.603 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-11T10:39:58.603 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2023-10-11T10:39:58.603 INFO:teuthology.orchestra.run.smithi050.stdout:======================================================================================================================= 2023-10-11T10:39:58.604 INFO:teuthology.orchestra.run.smithi050.stdout:Install 8 Packages 2023-10-11T10:39:58.604 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-11T10:39:58.604 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 35 M 2023-10-11T10:39:58.605 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 210 M 2023-10-11T10:39:58.605 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2023-10-11T10:39:58.710 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-6661.g4bf94f5 8/8 2023-10-11T10:39:58.710 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-6661.g4bf94f5 1/8 2023-10-11T10:39:58.710 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : libgfortran-8.5.0-10.1.el8_6.x86_64 2/8 2023-10-11T10:39:58.710 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : libquadmath-8.5.0-10.1.el8_6.x86_64 3/8 2023-10-11T10:39:58.711 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd434d 4/8 2023-10-11T10:39:58.711 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2023-10-11T10:39:58.711 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-10-11T10:39:58.711 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : openblas-threads-0.3.15-3.el8.x86_64 7/8 2023-10-11T10:39:58.786 INFO:teuthology.orchestra.run.smithi050.stdout:(1/8): libquadmath-8.5.0-10.1.el8_6.x86_64.rpm 940 kB/s | 171 kB 00:00 2023-10-11T10:39:58.827 INFO:teuthology.orchestra.run.smithi050.stdout:(2/8): libgfortran-8.5.0-10.1.el8_6.x86_64.rpm 2.8 MB/s | 644 kB 00:00 2023-10-11T10:39:59.019 INFO:teuthology.orchestra.run.smithi050.stdout:(3/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 19 MB/s | 3.7 MB 00:00 2023-10-11T10:39:59.228 INFO:teuthology.orchestra.run.smithi050.stdout:(4/8): python3-scipy-1.0.0-21.module+el8.5.0+10 31 MB/s | 14 MB 00:00 2023-10-11T10:39:59.245 INFO:teuthology.orchestra.run.smithi050.stdout:(5/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 998 kB/s | 225 kB 00:00 2023-10-11T10:39:59.479 INFO:teuthology.orchestra.run.smithi050.stdout:(6/8): openblas-threads-0.3.15-3.el8.x86_64.rpm 19 MB/s | 4.9 MB 00:00 2023-10-11T10:39:59.554 INFO:teuthology.orchestra.run.smithi050.stdout:(7/8): openblas-0.3.15-3.el8.x86_64.rpm 15 MB/s | 4.7 MB 00:00 2023-10-11T10:39:59.796 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : openblas-0.3.15-3.el8.x86_64 8/8 2023-10-11T10:39:59.796 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:02:20 ago on Wed 11 Oct 2023 10:37:39 AM UTC. 2023-10-11T10:39:59.863 INFO:teuthology.orchestra.run.smithi050.stdout:(8/8): ceph-mgr-diskprediction-local-18.0.0-666 5.9 MB/s | 7.4 MB 00:01 2023-10-11T10:39:59.864 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2023-10-11T10:39:59.864 INFO:teuthology.orchestra.run.smithi050.stdout:Total 28 MB/s | 35 MB 00:01 2023-10-11T10:40:00.110 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2023-10-11T10:40:00.129 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2023-10-11T10:40:00.129 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2023-10-11T10:40:00.438 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2023-10-11T10:40:00.442 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2023-10-11T10:40:00.994 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2023-10-11T10:40:01.154 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : libquadmath-8.5.0-10.1.el8_6.x86_64 1/8 2023-10-11T10:40:01.304 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: libquadmath-8.5.0-10.1.el8_6.x86_64 1/8 2023-10-11T10:40:01.479 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : libgfortran-8.5.0-10.1.el8_6.x86_64 2/8 2023-10-11T10:40:02.134 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: libgfortran-8.5.0-10.1.el8_6.x86_64 2/8 2023-10-11T10:40:02.309 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : openblas-threads-0.3.15-3.el8.x86_64 3/8 2023-10-11T10:40:02.949 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: openblas-threads-0.3.15-3.el8.x86_64 3/8 2023-10-11T10:40:03.095 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : openblas-0.3.15-3.el8.x86_64 4/8 2023-10-11T10:40:03.828 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: openblas-0.3.15-3.el8.x86_64 4/8 2023-10-11T10:40:04.119 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2023-10-11T10:40:04.288 INFO:teuthology.orchestra.run.smithi111.stdout:Installed products updated. 2023-10-11T10:40:04.320 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-10-11T10:40:04.474 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T10:40:04.474 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2023-10-11T10:40:04.475 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mgr-diskprediction-local-2:18.0.0-6661.g4bf94f59.el8.noarch 2023-10-11T10:40:04.475 INFO:teuthology.orchestra.run.smithi111.stdout: libgfortran-8.5.0-10.1.el8_6.x86_64 2023-10-11T10:40:04.475 INFO:teuthology.orchestra.run.smithi111.stdout: libquadmath-8.5.0-10.1.el8_6.x86_64 2023-10-11T10:40:04.475 INFO:teuthology.orchestra.run.smithi111.stdout: openblas-0.3.15-3.el8.x86_64 2023-10-11T10:40:04.475 INFO:teuthology.orchestra.run.smithi111.stdout: openblas-threads-0.3.15-3.el8.x86_64 2023-10-11T10:40:04.475 INFO:teuthology.orchestra.run.smithi111.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-10-11T10:40:04.476 INFO:teuthology.orchestra.run.smithi111.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-10-11T10:40:04.476 INFO:teuthology.orchestra.run.smithi111.stdout: python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd434d.x86_64 2023-10-11T10:40:04.476 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T10:40:04.476 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2023-10-11T10:40:04.614 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install ceph-mgr-rook 2023-10-11T10:40:04.960 INFO:teuthology.orchestra.run.smithi111.stdout:Updating Subscription Management repositories. 2023-10-11T10:40:06.192 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-10-11T10:40:06.369 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:02:27 ago on Wed 11 Oct 2023 10:37:39 AM UTC. 2023-10-11T10:40:07.467 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd434d 7/8 2023-10-11T10:40:07.619 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-6661.g4bf94f5 8/8 2023-10-11T10:40:07.828 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2023-10-11T10:40:07.830 INFO:teuthology.orchestra.run.smithi111.stdout:======================================================================================================== 2023-10-11T10:40:07.830 INFO:teuthology.orchestra.run.smithi111.stdout: Package Arch Version Repository Size 2023-10-11T10:40:07.831 INFO:teuthology.orchestra.run.smithi111.stdout:======================================================================================================== 2023-10-11T10:40:07.831 INFO:teuthology.orchestra.run.smithi111.stdout:Installing: 2023-10-11T10:40:07.831 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mgr-rook noarch 2:18.0.0-6661.g4bf94f59.el8 ceph-noarch 47 k 2023-10-11T10:40:07.831 INFO:teuthology.orchestra.run.smithi111.stdout:Installing dependencies: 2023-10-11T10:40:07.832 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-10-11T10:40:07.832 INFO:teuthology.orchestra.run.smithi111.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-10-11T10:40:07.832 INFO:teuthology.orchestra.run.smithi111.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-10-11T10:40:07.833 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2023-10-11T10:40:07.833 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2023-10-11T10:40:07.833 INFO:teuthology.orchestra.run.smithi111.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-10-11T10:40:07.833 INFO:teuthology.orchestra.run.smithi111.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2023-10-11T10:40:07.834 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2023-10-11T10:40:07.834 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2023-10-11T10:40:07.834 INFO:teuthology.orchestra.run.smithi111.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-10-11T10:40:07.835 INFO:teuthology.orchestra.run.smithi111.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2023-10-11T10:40:07.835 INFO:teuthology.orchestra.run.smithi111.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-10-11T10:40:07.835 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T10:40:07.835 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2023-10-11T10:40:07.836 INFO:teuthology.orchestra.run.smithi111.stdout:======================================================================================================== 2023-10-11T10:40:07.836 INFO:teuthology.orchestra.run.smithi111.stdout:Install 13 Packages 2023-10-11T10:40:07.836 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T10:40:07.837 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 1.9 M 2023-10-11T10:40:07.837 INFO:teuthology.orchestra.run.smithi111.stdout:Installed size: 25 M 2023-10-11T10:40:07.838 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2023-10-11T10:40:07.884 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-6661.g4bf94f5 8/8 2023-10-11T10:40:07.885 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-6661.g4bf94f5 1/8 2023-10-11T10:40:07.885 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : libgfortran-8.5.0-10.1.el8_6.x86_64 2/8 2023-10-11T10:40:07.885 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : libquadmath-8.5.0-10.1.el8_6.x86_64 3/8 2023-10-11T10:40:07.886 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd434d 4/8 2023-10-11T10:40:07.886 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2023-10-11T10:40:07.886 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-10-11T10:40:07.887 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : openblas-threads-0.3.15-3.el8.x86_64 7/8 2023-10-11T10:40:08.297 INFO:teuthology.orchestra.run.smithi111.stdout:(1/13): python3-certifi-2018.10.15-7.el8.noarch 169 kB/s | 16 kB 00:00 2023-10-11T10:40:08.323 INFO:teuthology.orchestra.run.smithi111.stdout:(2/13): python3-cachetools-3.1.1-4.el8.noarch.r 271 kB/s | 33 kB 00:00 2023-10-11T10:40:08.356 INFO:teuthology.orchestra.run.smithi111.stdout:(3/13): python3-google-auth-1.1.1-10.el8.noarch 1.4 MB/s | 81 kB 00:00 2023-10-11T10:40:08.406 INFO:teuthology.orchestra.run.smithi111.stdout:(4/13): python3-rsa-4.9-2.el8.noarch.rpm 1.3 MB/s | 64 kB 00:00 2023-10-11T10:40:08.432 INFO:teuthology.orchestra.run.smithi111.stdout:(5/13): ceph-mgr-rook-18.0.0-6661.g4bf94f59.el8 205 kB/s | 47 kB 00:00 2023-10-11T10:40:08.457 INFO:teuthology.orchestra.run.smithi111.stdout:(6/13): python3-websocket-client-0.56.0-5.el8.n 1.2 MB/s | 61 kB 00:00 2023-10-11T10:40:08.499 INFO:teuthology.orchestra.run.smithi111.stdout:(7/13): python3-kubernetes-11.0.0-6.el8.noarch. 6.6 MB/s | 1.2 MB 00:00 2023-10-11T10:40:08.608 INFO:teuthology.orchestra.run.smithi111.stdout:(8/13): python3-requests-oauthlib-1.0.0-1.el8.n 243 kB/s | 43 kB 00:00 2023-10-11T10:40:08.633 INFO:teuthology.orchestra.run.smithi111.stdout:(9/13): python3-pyasn1-modules-0.3.7-6.el8.noar 898 kB/s | 110 kB 00:00 2023-10-11T10:40:08.658 INFO:teuthology.orchestra.run.smithi111.stdout:(10/13): python3-jsonpatch-1.21-2.el8.noarch.rp 529 kB/s | 27 kB 00:00 2023-10-11T10:40:08.684 INFO:teuthology.orchestra.run.smithi111.stdout:(11/13): python3-oauthlib-2.1.0-1.el8.noarch.rp 684 kB/s | 155 kB 00:00 2023-10-11T10:40:08.709 INFO:teuthology.orchestra.run.smithi111.stdout:(12/13): python3-jsonpointer-1.10-11.el8.noarch 398 kB/s | 20 kB 00:00 2023-10-11T10:40:08.734 INFO:teuthology.orchestra.run.smithi111.stdout:(13/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 1.2 MB/s | 126 kB 00:00 2023-10-11T10:40:08.735 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2023-10-11T10:40:08.735 INFO:teuthology.orchestra.run.smithi111.stdout:Total 2.1 MB/s | 1.9 MB 00:00 2023-10-11T10:40:08.800 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2023-10-11T10:40:08.816 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2023-10-11T10:40:08.816 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2023-10-11T10:40:08.886 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2023-10-11T10:40:08.890 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2023-10-11T10:40:08.987 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : openblas-0.3.15-3.el8.x86_64 8/8 2023-10-11T10:40:08.987 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:02:27 ago on Wed 11 Oct 2023 10:37:41 AM UTC. 2023-10-11T10:40:09.395 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2023-10-11T10:40:09.648 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2023-10-11T10:40:09.900 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-rsa-4.9-2.el8.noarch 2/13 2023-10-11T10:40:10.054 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 3/13 2023-10-11T10:40:10.238 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 4/13 2023-10-11T10:40:10.475 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 5/13 2023-10-11T10:40:10.706 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2023-10-11T10:40:10.909 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2023-10-11T10:40:11.121 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 8/13 2023-10-11T10:40:11.270 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 9/13 2023-10-11T10:40:11.486 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 10/13 2023-10-11T10:40:12.139 INFO:teuthology.orchestra.run.smithi050.stdout:Installed products updated. 2023-10-11T10:40:12.339 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-11T10:40:12.339 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2023-10-11T10:40:12.339 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-diskprediction-local-2:18.0.0-6661.g4bf94f59.el8.noarch 2023-10-11T10:40:12.340 INFO:teuthology.orchestra.run.smithi050.stdout: libgfortran-8.5.0-10.1.el8_6.x86_64 2023-10-11T10:40:12.340 INFO:teuthology.orchestra.run.smithi050.stdout: libquadmath-8.5.0-10.1.el8_6.x86_64 2023-10-11T10:40:12.340 INFO:teuthology.orchestra.run.smithi050.stdout: openblas-0.3.15-3.el8.x86_64 2023-10-11T10:40:12.340 INFO:teuthology.orchestra.run.smithi050.stdout: openblas-threads-0.3.15-3.el8.x86_64 2023-10-11T10:40:12.340 INFO:teuthology.orchestra.run.smithi050.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-10-11T10:40:12.341 INFO:teuthology.orchestra.run.smithi050.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-10-11T10:40:12.341 INFO:teuthology.orchestra.run.smithi050.stdout: python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd434d.x86_64 2023-10-11T10:40:12.341 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-11T10:40:12.341 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2023-10-11T10:40:12.481 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install ceph-mgr-rook 2023-10-11T10:40:12.574 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 11/13 2023-10-11T10:40:12.789 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 12/13 2023-10-11T10:40:12.850 INFO:teuthology.orchestra.run.smithi050.stdout:Updating Subscription Management repositories. 2023-10-11T10:40:12.956 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-mgr-rook-2:18.0.0-6661.g4bf94f59.el8.noarch 13/13 2023-10-11T10:40:13.199 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-6661.g4bf94f59.el8.noarch 13/13 2023-10-11T10:40:13.199 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-mgr-rook-2:18.0.0-6661.g4bf94f59.el8.noarch 1/13 2023-10-11T10:40:13.199 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 2/13 2023-10-11T10:40:13.199 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 3/13 2023-10-11T10:40:13.200 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 4/13 2023-10-11T10:40:13.200 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 5/13 2023-10-11T10:40:13.200 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 6/13 2023-10-11T10:40:13.200 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 7/13 2023-10-11T10:40:13.200 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 8/13 2023-10-11T10:40:13.201 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 9/13 2023-10-11T10:40:13.201 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 10/13 2023-10-11T10:40:13.201 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 11/13 2023-10-11T10:40:13.201 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 12/13 2023-10-11T10:40:14.311 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 13/13 2023-10-11T10:40:14.311 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:02:35 ago on Wed 11 Oct 2023 10:37:39 AM UTC. 2023-10-11T10:40:14.386 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:02:33 ago on Wed 11 Oct 2023 10:37:41 AM UTC. 2023-10-11T10:40:15.957 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2023-10-11T10:40:15.959 INFO:teuthology.orchestra.run.smithi050.stdout:======================================================================================================== 2023-10-11T10:40:15.959 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repository Size 2023-10-11T10:40:15.959 INFO:teuthology.orchestra.run.smithi050.stdout:======================================================================================================== 2023-10-11T10:40:15.960 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2023-10-11T10:40:15.960 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-rook noarch 2:18.0.0-6661.g4bf94f59.el8 ceph-noarch 47 k 2023-10-11T10:40:15.960 INFO:teuthology.orchestra.run.smithi050.stdout:Installing dependencies: 2023-10-11T10:40:15.960 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-10-11T10:40:15.960 INFO:teuthology.orchestra.run.smithi050.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-10-11T10:40:15.961 INFO:teuthology.orchestra.run.smithi050.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-10-11T10:40:15.961 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2023-10-11T10:40:15.961 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2023-10-11T10:40:15.961 INFO:teuthology.orchestra.run.smithi050.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-10-11T10:40:15.961 INFO:teuthology.orchestra.run.smithi050.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2023-10-11T10:40:15.962 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2023-10-11T10:40:15.962 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2023-10-11T10:40:15.962 INFO:teuthology.orchestra.run.smithi050.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-10-11T10:40:15.962 INFO:teuthology.orchestra.run.smithi050.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2023-10-11T10:40:15.962 INFO:teuthology.orchestra.run.smithi050.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-10-11T10:40:15.963 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-11T10:40:15.963 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2023-10-11T10:40:15.963 INFO:teuthology.orchestra.run.smithi050.stdout:======================================================================================================== 2023-10-11T10:40:15.963 INFO:teuthology.orchestra.run.smithi050.stdout:Install 13 Packages 2023-10-11T10:40:15.963 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-11T10:40:15.964 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 1.9 M 2023-10-11T10:40:15.964 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 25 M 2023-10-11T10:40:15.964 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2023-10-11T10:40:16.477 INFO:teuthology.orchestra.run.smithi050.stdout:(1/13): python3-certifi-2018.10.15-7.el8.noarch 121 kB/s | 16 kB 00:00 2023-10-11T10:40:16.502 INFO:teuthology.orchestra.run.smithi050.stdout:(2/13): python3-cachetools-3.1.1-4.el8.noarch.r 206 kB/s | 33 kB 00:00 2023-10-11T10:40:16.594 INFO:teuthology.orchestra.run.smithi050.stdout:(3/13): python3-google-auth-1.1.1-10.el8.noarch 694 kB/s | 81 kB 00:00 2023-10-11T10:40:16.619 INFO:teuthology.orchestra.run.smithi050.stdout:(4/13): ceph-mgr-rook-18.0.0-6661.g4bf94f59.el8 171 kB/s | 47 kB 00:00 2023-10-11T10:40:16.678 INFO:teuthology.orchestra.run.smithi050.stdout:(5/13): python3-rsa-4.9-2.el8.noarch.rpm 770 kB/s | 64 kB 00:00 2023-10-11T10:40:16.787 INFO:teuthology.orchestra.run.smithi050.stdout:(6/13): python3-requests-oauthlib-1.0.0-1.el8.n 394 kB/s | 43 kB 00:00 2023-10-11T10:40:16.837 INFO:teuthology.orchestra.run.smithi050.stdout:(7/13): python3-kubernetes-11.0.0-6.el8.noarch. 3.5 MB/s | 1.2 MB 00:00 2023-10-11T10:40:16.862 INFO:teuthology.orchestra.run.smithi050.stdout:(8/13): python3-websocket-client-0.56.0-5.el8.n 252 kB/s | 61 kB 00:00 2023-10-11T10:40:16.954 INFO:teuthology.orchestra.run.smithi050.stdout:(9/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 929 kB/s | 155 kB 00:00 2023-10-11T10:40:17.004 INFO:teuthology.orchestra.run.smithi050.stdout:(10/13): python3-jsonpatch-1.21-2.el8.noarch.rp 199 kB/s | 27 kB 00:00 2023-10-11T10:40:17.029 INFO:teuthology.orchestra.run.smithi050.stdout:(11/13): python3-pyasn1-modules-0.3.7-6.el8.noa 572 kB/s | 110 kB 00:00 2023-10-11T10:40:17.080 INFO:teuthology.orchestra.run.smithi050.stdout:(12/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 1.0 MB/s | 126 kB 00:00 2023-10-11T10:40:17.197 INFO:teuthology.orchestra.run.smithi050.stdout:(13/13): python3-jsonpointer-1.10-11.el8.noarch 104 kB/s | 20 kB 00:00 2023-10-11T10:40:17.197 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2023-10-11T10:40:17.198 INFO:teuthology.orchestra.run.smithi050.stdout:Total 1.6 MB/s | 1.9 MB 00:01 2023-10-11T10:40:17.268 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2023-10-11T10:40:17.286 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2023-10-11T10:40:17.286 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2023-10-11T10:40:17.363 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2023-10-11T10:40:17.367 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2023-10-11T10:40:17.863 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2023-10-11T10:40:18.070 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2023-10-11T10:40:18.299 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-rsa-4.9-2.el8.noarch 2/13 2023-10-11T10:40:18.483 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 3/13 2023-10-11T10:40:18.634 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 4/13 2023-10-11T10:40:18.854 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 5/13 2023-10-11T10:40:19.066 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2023-10-11T10:40:19.282 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2023-10-11T10:40:19.467 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 8/13 2023-10-11T10:40:19.626 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 9/13 2023-10-11T10:40:19.837 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 10/13 2023-10-11T10:40:19.983 INFO:teuthology.orchestra.run.smithi111.stdout:Installed products updated. 2023-10-11T10:40:20.170 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T10:40:20.171 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2023-10-11T10:40:20.171 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mgr-rook-2:18.0.0-6661.g4bf94f59.el8.noarch 2023-10-11T10:40:20.171 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-10-11T10:40:20.171 INFO:teuthology.orchestra.run.smithi111.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-10-11T10:40:20.172 INFO:teuthology.orchestra.run.smithi111.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-10-11T10:40:20.172 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-10-11T10:40:20.172 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-10-11T10:40:20.172 INFO:teuthology.orchestra.run.smithi111.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-10-11T10:40:20.172 INFO:teuthology.orchestra.run.smithi111.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-10-11T10:40:20.172 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-10-11T10:40:20.173 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-10-11T10:40:20.173 INFO:teuthology.orchestra.run.smithi111.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-10-11T10:40:20.173 INFO:teuthology.orchestra.run.smithi111.stdout: python3-rsa-4.9-2.el8.noarch 2023-10-11T10:40:20.173 INFO:teuthology.orchestra.run.smithi111.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-10-11T10:40:20.173 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T10:40:20.174 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2023-10-11T10:40:20.314 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install ceph-mgr-cephadm 2023-10-11T10:40:20.664 INFO:teuthology.orchestra.run.smithi111.stdout:Updating Subscription Management repositories. 2023-10-11T10:40:20.976 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 11/13 2023-10-11T10:40:21.345 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 12/13 2023-10-11T10:40:21.485 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-mgr-rook-2:18.0.0-6661.g4bf94f59.el8.noarch 13/13 2023-10-11T10:40:21.529 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:02:42 ago on Wed 11 Oct 2023 10:37:39 AM UTC. 2023-10-11T10:40:21.759 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-6661.g4bf94f59.el8.noarch 13/13 2023-10-11T10:40:21.759 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-mgr-rook-2:18.0.0-6661.g4bf94f59.el8.noarch 1/13 2023-10-11T10:40:21.760 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 2/13 2023-10-11T10:40:21.760 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 3/13 2023-10-11T10:40:21.760 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 4/13 2023-10-11T10:40:21.760 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 5/13 2023-10-11T10:40:21.760 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 6/13 2023-10-11T10:40:21.761 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 7/13 2023-10-11T10:40:21.761 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 8/13 2023-10-11T10:40:21.761 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 9/13 2023-10-11T10:40:21.761 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 10/13 2023-10-11T10:40:21.761 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 11/13 2023-10-11T10:40:21.762 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 12/13 2023-10-11T10:40:22.893 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 13/13 2023-10-11T10:40:22.893 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:02:41 ago on Wed 11 Oct 2023 10:37:41 AM UTC. 2023-10-11T10:40:22.917 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2023-10-11T10:40:22.918 INFO:teuthology.orchestra.run.smithi111.stdout:=============================================================================================== 2023-10-11T10:40:22.918 INFO:teuthology.orchestra.run.smithi111.stdout: Package Arch Version Repository Size 2023-10-11T10:40:22.918 INFO:teuthology.orchestra.run.smithi111.stdout:=============================================================================================== 2023-10-11T10:40:22.919 INFO:teuthology.orchestra.run.smithi111.stdout:Installing: 2023-10-11T10:40:22.919 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mgr-cephadm noarch 2:18.0.0-6661.g4bf94f59.el8 ceph-noarch 132 k 2023-10-11T10:40:22.919 INFO:teuthology.orchestra.run.smithi111.stdout:Installing dependencies: 2023-10-11T10:40:22.919 INFO:teuthology.orchestra.run.smithi111.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-10-11T10:40:22.919 INFO:teuthology.orchestra.run.smithi111.stdout: python3-babel noarch 2.5.1-7.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2023-10-11T10:40:22.920 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jinja2 noarch 2.10.1-3.el8 rhel-8-for-x86_64-appstream-rpms 538 k 2023-10-11T10:40:22.920 INFO:teuthology.orchestra.run.smithi111.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-10-11T10:40:22.920 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T10:40:22.920 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2023-10-11T10:40:22.920 INFO:teuthology.orchestra.run.smithi111.stdout:=============================================================================================== 2023-10-11T10:40:22.921 INFO:teuthology.orchestra.run.smithi111.stdout:Install 5 Packages 2023-10-11T10:40:22.921 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T10:40:22.921 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 6.0 M 2023-10-11T10:40:22.922 INFO:teuthology.orchestra.run.smithi111.stdout:Installed size: 26 M 2023-10-11T10:40:22.922 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2023-10-11T10:40:23.118 INFO:teuthology.orchestra.run.smithi111.stdout:(1/5): python3-natsort-7.1.1-2.el8.noarch.rpm 439 kB/s | 60 kB 00:00 2023-10-11T10:40:23.194 INFO:teuthology.orchestra.run.smithi111.stdout:(2/5): python3-jinja2-2.10.1-3.el8.noarch.rpm 7.0 MB/s | 538 kB 00:00 2023-10-11T10:40:23.219 INFO:teuthology.orchestra.run.smithi111.stdout:(3/5): python3-asyncssh-2.7.0-2.el8.noarch.rpm 2.0 MB/s | 476 kB 00:00 2023-10-11T10:40:23.244 INFO:teuthology.orchestra.run.smithi111.stdout:(4/5): ceph-mgr-cephadm-18.0.0-6661.g4bf94f59.e 501 kB/s | 132 kB 00:00 2023-10-11T10:40:23.660 INFO:teuthology.orchestra.run.smithi111.stdout:(5/5): python3-babel-2.5.1-7.el8.noarch.rpm 10 MB/s | 4.8 MB 00:00 2023-10-11T10:40:23.661 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2023-10-11T10:40:23.661 INFO:teuthology.orchestra.run.smithi111.stdout:Total 8.0 MB/s | 6.0 MB 00:00 2023-10-11T10:40:23.717 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2023-10-11T10:40:23.728 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2023-10-11T10:40:23.728 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2023-10-11T10:40:23.873 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2023-10-11T10:40:23.877 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2023-10-11T10:40:24.976 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2023-10-11T10:40:25.340 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 1/5 2023-10-11T10:40:25.555 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 2/5 2023-10-11T10:40:25.908 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 3/5 2023-10-11T10:40:26.169 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 4/5 2023-10-11T10:40:26.412 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-6661.g4bf94f59.el8.noarch 5/5 2023-10-11T10:40:26.646 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-6661.g4bf94f59.el8.noarch 5/5 2023-10-11T10:40:26.646 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-6661.g4bf94f59.el8.noarch 1/5 2023-10-11T10:40:26.647 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 2/5 2023-10-11T10:40:26.647 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 3/5 2023-10-11T10:40:26.647 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 4/5 2023-10-11T10:40:27.676 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 5/5 2023-10-11T10:40:27.677 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:02:48 ago on Wed 11 Oct 2023 10:37:39 AM UTC. 2023-10-11T10:40:28.293 INFO:teuthology.orchestra.run.smithi050.stdout:Installed products updated. 2023-10-11T10:40:28.493 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-11T10:40:28.493 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2023-10-11T10:40:28.493 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-rook-2:18.0.0-6661.g4bf94f59.el8.noarch 2023-10-11T10:40:28.493 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-10-11T10:40:28.494 INFO:teuthology.orchestra.run.smithi050.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-10-11T10:40:28.494 INFO:teuthology.orchestra.run.smithi050.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-10-11T10:40:28.494 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-10-11T10:40:28.494 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-10-11T10:40:28.494 INFO:teuthology.orchestra.run.smithi050.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-10-11T10:40:28.495 INFO:teuthology.orchestra.run.smithi050.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-10-11T10:40:28.495 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-10-11T10:40:28.495 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-10-11T10:40:28.495 INFO:teuthology.orchestra.run.smithi050.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-10-11T10:40:28.495 INFO:teuthology.orchestra.run.smithi050.stdout: python3-rsa-4.9-2.el8.noarch 2023-10-11T10:40:28.496 INFO:teuthology.orchestra.run.smithi050.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-10-11T10:40:28.496 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-11T10:40:28.496 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2023-10-11T10:40:28.642 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install ceph-mgr-cephadm 2023-10-11T10:40:29.015 INFO:teuthology.orchestra.run.smithi050.stdout:Updating Subscription Management repositories. 2023-10-11T10:40:30.374 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:02:49 ago on Wed 11 Oct 2023 10:37:41 AM UTC. 2023-10-11T10:40:31.922 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2023-10-11T10:40:31.923 INFO:teuthology.orchestra.run.smithi050.stdout:=============================================================================================== 2023-10-11T10:40:31.923 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repository Size 2023-10-11T10:40:31.924 INFO:teuthology.orchestra.run.smithi050.stdout:=============================================================================================== 2023-10-11T10:40:31.924 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2023-10-11T10:40:31.925 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-cephadm noarch 2:18.0.0-6661.g4bf94f59.el8 ceph-noarch 132 k 2023-10-11T10:40:31.925 INFO:teuthology.orchestra.run.smithi050.stdout:Installing dependencies: 2023-10-11T10:40:31.925 INFO:teuthology.orchestra.run.smithi050.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-10-11T10:40:31.925 INFO:teuthology.orchestra.run.smithi050.stdout: python3-babel noarch 2.5.1-7.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2023-10-11T10:40:31.926 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jinja2 noarch 2.10.1-3.el8 rhel-8-for-x86_64-appstream-rpms 538 k 2023-10-11T10:40:31.926 INFO:teuthology.orchestra.run.smithi050.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-10-11T10:40:31.926 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-11T10:40:31.926 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2023-10-11T10:40:31.926 INFO:teuthology.orchestra.run.smithi050.stdout:=============================================================================================== 2023-10-11T10:40:31.927 INFO:teuthology.orchestra.run.smithi050.stdout:Install 5 Packages 2023-10-11T10:40:31.927 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-11T10:40:31.927 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 6.0 M 2023-10-11T10:40:31.928 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 26 M 2023-10-11T10:40:31.928 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2023-10-11T10:40:32.071 INFO:teuthology.orchestra.run.smithi111.stdout:Installed products updated. 2023-10-11T10:40:32.254 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T10:40:32.255 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2023-10-11T10:40:32.255 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mgr-cephadm-2:18.0.0-6661.g4bf94f59.el8.noarch 2023-10-11T10:40:32.255 INFO:teuthology.orchestra.run.smithi111.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-10-11T10:40:32.255 INFO:teuthology.orchestra.run.smithi111.stdout: python3-babel-2.5.1-7.el8.noarch 2023-10-11T10:40:32.255 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jinja2-2.10.1-3.el8.noarch 2023-10-11T10:40:32.256 INFO:teuthology.orchestra.run.smithi111.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-10-11T10:40:32.256 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T10:40:32.256 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2023-10-11T10:40:32.379 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install ceph-fuse 2023-10-11T10:40:32.546 INFO:teuthology.orchestra.run.smithi050.stdout:(1/5): python3-natsort-7.1.1-2.el8.noarch.rpm 223 kB/s | 60 kB 00:00 2023-10-11T10:40:32.604 INFO:teuthology.orchestra.run.smithi050.stdout:(2/5): ceph-mgr-cephadm-18.0.0-6661.g4bf94f59.e 402 kB/s | 132 kB 00:00 2023-10-11T10:40:32.679 INFO:teuthology.orchestra.run.smithi050.stdout:(3/5): python3-asyncssh-2.7.0-2.el8.noarch.rpm 1.2 MB/s | 476 kB 00:00 2023-10-11T10:40:32.730 INFO:teuthology.orchestra.run.smithi111.stdout:Updating Subscription Management repositories. 2023-10-11T10:40:32.738 INFO:teuthology.orchestra.run.smithi050.stdout:(4/5): python3-jinja2-2.10.1-3.el8.noarch.rpm 2.7 MB/s | 538 kB 00:00 2023-10-11T10:40:32.930 INFO:teuthology.orchestra.run.smithi050.stdout:(5/5): python3-babel-2.5.1-7.el8.noarch.rpm 15 MB/s | 4.8 MB 00:00 2023-10-11T10:40:32.932 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2023-10-11T10:40:32.932 INFO:teuthology.orchestra.run.smithi050.stdout:Total 5.9 MB/s | 6.0 MB 00:01 2023-10-11T10:40:32.990 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2023-10-11T10:40:33.002 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2023-10-11T10:40:33.002 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2023-10-11T10:40:33.086 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2023-10-11T10:40:33.090 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2023-10-11T10:40:34.141 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2023-10-11T10:40:34.265 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:02:55 ago on Wed 11 Oct 2023 10:37:39 AM UTC. 2023-10-11T10:40:34.504 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 1/5 2023-10-11T10:40:34.702 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 2/5 2023-10-11T10:40:35.004 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 3/5 2023-10-11T10:40:35.284 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 4/5 2023-10-11T10:40:35.466 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-6661.g4bf94f59.el8.noarch 5/5 2023-10-11T10:40:35.648 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2023-10-11T10:40:35.648 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2023-10-11T10:40:35.649 INFO:teuthology.orchestra.run.smithi111.stdout: Package Arch Version Repository Size 2023-10-11T10:40:35.649 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2023-10-11T10:40:35.649 INFO:teuthology.orchestra.run.smithi111.stdout:Installing: 2023-10-11T10:40:35.649 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-fuse 2023-10-11T10:40:35.649 INFO:teuthology.orchestra.run.smithi111.stdout: x86_64 2:18.0.0-6661.g4bf94f59.el8 ceph 918 k 2023-10-11T10:40:35.650 INFO:teuthology.orchestra.run.smithi111.stdout:Installing dependencies: 2023-10-11T10:40:35.650 INFO:teuthology.orchestra.run.smithi111.stdout: fuse x86_64 2.9.7-15.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2023-10-11T10:40:35.650 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T10:40:35.650 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2023-10-11T10:40:35.650 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2023-10-11T10:40:35.651 INFO:teuthology.orchestra.run.smithi111.stdout:Install 2 Packages 2023-10-11T10:40:35.651 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T10:40:35.651 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 1.0 M 2023-10-11T10:40:35.652 INFO:teuthology.orchestra.run.smithi111.stdout:Installed size: 3.1 M 2023-10-11T10:40:35.652 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2023-10-11T10:40:35.707 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-6661.g4bf94f59.el8.noarch 5/5 2023-10-11T10:40:35.707 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-6661.g4bf94f59.el8.noarch 1/5 2023-10-11T10:40:35.707 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 2/5 2023-10-11T10:40:35.708 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 3/5 2023-10-11T10:40:35.708 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 4/5 2023-10-11T10:40:35.806 INFO:teuthology.orchestra.run.smithi111.stdout:(1/2): fuse-2.9.7-15.el8.x86_64.rpm 531 kB/s | 83 kB 00:00 2023-10-11T10:40:35.923 INFO:teuthology.orchestra.run.smithi111.stdout:(2/2): ceph-fuse-18.0.0-6661.g4bf94f59.el8.x86_ 3.3 MB/s | 918 kB 00:00 2023-10-11T10:40:35.924 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2023-10-11T10:40:35.925 INFO:teuthology.orchestra.run.smithi111.stdout:Total 3.6 MB/s | 1.0 MB 00:00 2023-10-11T10:40:35.936 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2023-10-11T10:40:35.958 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2023-10-11T10:40:35.958 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2023-10-11T10:40:36.031 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2023-10-11T10:40:36.035 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2023-10-11T10:40:36.211 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2023-10-11T10:40:36.473 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : fuse-2.9.7-15.el8.x86_64 1/2 2023-10-11T10:40:36.633 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-fuse-2:18.0.0-6661.g4bf94f59.el8.x86_64 2/2 2023-10-11T10:40:36.794 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 5/5 2023-10-11T10:40:36.795 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:02:55 ago on Wed 11 Oct 2023 10:37:41 AM UTC. 2023-10-11T10:40:36.970 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-fuse-2:18.0.0-6661.g4bf94f59.el8.x86_64 2/2 2023-10-11T10:40:36.970 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-fuse-2:18.0.0-6661.g4bf94f59.el8.x86_64 1/2 2023-10-11T10:40:38.023 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : fuse-2.9.7-15.el8.x86_64 2/2 2023-10-11T10:40:38.024 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:02:59 ago on Wed 11 Oct 2023 10:37:39 AM UTC. 2023-10-11T10:40:43.377 INFO:teuthology.orchestra.run.smithi111.stdout:Installed products updated. 2023-10-11T10:40:43.560 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T10:40:43.560 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2023-10-11T10:40:43.560 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-fuse-2:18.0.0-6661.g4bf94f59.el8.x86_64 fuse-2.9.7-15.el8.x86_64 2023-10-11T10:40:43.560 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T10:40:43.561 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2023-10-11T10:40:43.576 INFO:teuthology.orchestra.run.smithi050.stdout:Installed products updated. 2023-10-11T10:40:43.683 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install ceph-volume 2023-10-11T10:40:43.776 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-11T10:40:43.776 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2023-10-11T10:40:43.776 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-cephadm-2:18.0.0-6661.g4bf94f59.el8.noarch 2023-10-11T10:40:43.776 INFO:teuthology.orchestra.run.smithi050.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-10-11T10:40:43.777 INFO:teuthology.orchestra.run.smithi050.stdout: python3-babel-2.5.1-7.el8.noarch 2023-10-11T10:40:43.777 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jinja2-2.10.1-3.el8.noarch 2023-10-11T10:40:43.777 INFO:teuthology.orchestra.run.smithi050.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-10-11T10:40:43.777 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-11T10:40:43.777 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2023-10-11T10:40:43.902 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install ceph-fuse 2023-10-11T10:40:44.037 INFO:teuthology.orchestra.run.smithi111.stdout:Updating Subscription Management repositories. 2023-10-11T10:40:44.277 INFO:teuthology.orchestra.run.smithi050.stdout:Updating Subscription Management repositories. 2023-10-11T10:40:45.503 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:03:06 ago on Wed 11 Oct 2023 10:37:39 AM UTC. 2023-10-11T10:40:45.818 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:03:04 ago on Wed 11 Oct 2023 10:37:41 AM UTC. 2023-10-11T10:40:46.905 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2023-10-11T10:40:46.906 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2023-10-11T10:40:46.906 INFO:teuthology.orchestra.run.smithi111.stdout: Package Arch Version Repository Size 2023-10-11T10:40:46.907 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2023-10-11T10:40:46.907 INFO:teuthology.orchestra.run.smithi111.stdout:Installing: 2023-10-11T10:40:46.907 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-volume noarch 2:18.0.0-6661.g4bf94f59.el8 ceph-noarch 261 k 2023-10-11T10:40:46.907 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T10:40:46.907 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2023-10-11T10:40:46.907 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2023-10-11T10:40:46.908 INFO:teuthology.orchestra.run.smithi111.stdout:Install 1 Package 2023-10-11T10:40:46.908 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T10:40:46.908 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 261 k 2023-10-11T10:40:46.908 INFO:teuthology.orchestra.run.smithi111.stdout:Installed size: 1.3 M 2023-10-11T10:40:46.908 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2023-10-11T10:40:47.121 INFO:teuthology.orchestra.run.smithi111.stdout:ceph-volume-18.0.0-6661.g4bf94f59.el8.noarch.rp 1.2 MB/s | 261 kB 00:00 2023-10-11T10:40:47.122 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2023-10-11T10:40:47.122 INFO:teuthology.orchestra.run.smithi111.stdout:Total 1.2 MB/s | 261 kB 00:00 2023-10-11T10:40:47.123 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2023-10-11T10:40:47.133 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2023-10-11T10:40:47.133 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2023-10-11T10:40:47.149 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2023-10-11T10:40:47.153 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2023-10-11T10:40:47.322 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2023-10-11T10:40:47.360 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2023-10-11T10:40:47.360 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-10-11T10:40:47.360 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repository Size 2023-10-11T10:40:47.361 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-10-11T10:40:47.361 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2023-10-11T10:40:47.361 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-fuse 2023-10-11T10:40:47.361 INFO:teuthology.orchestra.run.smithi050.stdout: x86_64 2:18.0.0-6661.g4bf94f59.el8 ceph 918 k 2023-10-11T10:40:47.361 INFO:teuthology.orchestra.run.smithi050.stdout:Installing dependencies: 2023-10-11T10:40:47.362 INFO:teuthology.orchestra.run.smithi050.stdout: fuse x86_64 2.9.7-15.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2023-10-11T10:40:47.362 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-11T10:40:47.362 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2023-10-11T10:40:47.362 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-10-11T10:40:47.362 INFO:teuthology.orchestra.run.smithi050.stdout:Install 2 Packages 2023-10-11T10:40:47.362 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-11T10:40:47.363 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 1.0 M 2023-10-11T10:40:47.363 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 3.1 M 2023-10-11T10:40:47.363 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2023-10-11T10:40:47.532 INFO:teuthology.orchestra.run.smithi050.stdout:(1/2): fuse-2.9.7-15.el8.x86_64.rpm 485 kB/s | 83 kB 00:00 2023-10-11T10:40:47.592 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-volume-2:18.0.0-6661.g4bf94f59.el8.noarch 1/1 2023-10-11T10:40:47.641 INFO:teuthology.orchestra.run.smithi050.stdout:(2/2): ceph-fuse-18.0.0-6661.g4bf94f59.el8.x86_ 3.2 MB/s | 918 kB 00:00 2023-10-11T10:40:47.642 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2023-10-11T10:40:47.642 INFO:teuthology.orchestra.run.smithi050.stdout:Total 3.5 MB/s | 1.0 MB 00:00 2023-10-11T10:40:47.654 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2023-10-11T10:40:47.677 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2023-10-11T10:40:47.677 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2023-10-11T10:40:47.756 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2023-10-11T10:40:47.760 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2023-10-11T10:40:47.929 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-volume-2:18.0.0-6661.g4bf94f59.el8.noarch 1/1 2023-10-11T10:40:47.930 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2023-10-11T10:40:48.190 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : fuse-2.9.7-15.el8.x86_64 1/2 2023-10-11T10:40:48.625 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-fuse-2:18.0.0-6661.g4bf94f59.el8.x86_64 2/2 2023-10-11T10:40:48.937 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-volume-2:18.0.0-6661.g4bf94f59.el8.noarch 1/1 2023-10-11T10:40:48.937 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:03:09 ago on Wed 11 Oct 2023 10:37:39 AM UTC. 2023-10-11T10:40:48.992 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-fuse-2:18.0.0-6661.g4bf94f59.el8.x86_64 2/2 2023-10-11T10:40:48.992 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-fuse-2:18.0.0-6661.g4bf94f59.el8.x86_64 1/2 2023-10-11T10:40:50.067 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : fuse-2.9.7-15.el8.x86_64 2/2 2023-10-11T10:40:50.068 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:03:09 ago on Wed 11 Oct 2023 10:37:41 AM UTC. 2023-10-11T10:40:53.449 INFO:teuthology.orchestra.run.smithi111.stdout:Installed products updated. 2023-10-11T10:40:53.628 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T10:40:53.628 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2023-10-11T10:40:53.629 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-volume-2:18.0.0-6661.g4bf94f59.el8.noarch 2023-10-11T10:40:53.629 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T10:40:53.629 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2023-10-11T10:40:53.760 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install librados-devel 2023-10-11T10:40:54.115 INFO:teuthology.orchestra.run.smithi111.stdout:Updating Subscription Management repositories. 2023-10-11T10:40:55.714 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:03:16 ago on Wed 11 Oct 2023 10:37:39 AM UTC. 2023-10-11T10:40:57.100 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2023-10-11T10:40:57.101 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2023-10-11T10:40:57.102 INFO:teuthology.orchestra.run.smithi111.stdout: Package Arch Version Repo Size 2023-10-11T10:40:57.102 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2023-10-11T10:40:57.102 INFO:teuthology.orchestra.run.smithi111.stdout:Installing: 2023-10-11T10:40:57.102 INFO:teuthology.orchestra.run.smithi111.stdout: librados-devel x86_64 2:18.0.0-6661.g4bf94f59.el8 ceph 139 k 2023-10-11T10:40:57.102 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T10:40:57.103 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2023-10-11T10:40:57.103 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2023-10-11T10:40:57.103 INFO:teuthology.orchestra.run.smithi111.stdout:Install 1 Package 2023-10-11T10:40:57.103 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T10:40:57.103 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 139 k 2023-10-11T10:40:57.104 INFO:teuthology.orchestra.run.smithi111.stdout:Installed size: 513 k 2023-10-11T10:40:57.104 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2023-10-11T10:40:57.294 INFO:teuthology.orchestra.run.smithi111.stdout:librados-devel-18.0.0-6661.g4bf94f59.el8.x86_64 723 kB/s | 139 kB 00:00 2023-10-11T10:40:57.295 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2023-10-11T10:40:57.296 INFO:teuthology.orchestra.run.smithi111.stdout:Total 719 kB/s | 139 kB 00:00 2023-10-11T10:40:57.296 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2023-10-11T10:40:57.301 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2023-10-11T10:40:57.301 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2023-10-11T10:40:57.354 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2023-10-11T10:40:57.358 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2023-10-11T10:40:57.468 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2023-10-11T10:40:57.646 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : librados-devel-2:18.0.0-6661.g4bf94f59.el8.x86_64 1/1 2023-10-11T10:40:57.882 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: librados-devel-2:18.0.0-6661.g4bf94f59.el8.x86_64 1/1 2023-10-11T10:40:58.770 INFO:teuthology.orchestra.run.smithi050.stdout:Installed products updated. 2023-10-11T10:40:58.966 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : librados-devel-2:18.0.0-6661.g4bf94f59.el8.x86_64 1/1 2023-10-11T10:40:58.966 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:03:19 ago on Wed 11 Oct 2023 10:37:39 AM UTC. 2023-10-11T10:40:58.969 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-11T10:40:58.969 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2023-10-11T10:40:58.970 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-fuse-2:18.0.0-6661.g4bf94f59.el8.x86_64 fuse-2.9.7-15.el8.x86_64 2023-10-11T10:40:58.970 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-11T10:40:58.970 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2023-10-11T10:40:59.107 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install ceph-volume 2023-10-11T10:40:59.484 INFO:teuthology.orchestra.run.smithi050.stdout:Updating Subscription Management repositories. 2023-10-11T10:41:00.380 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:03:19 ago on Wed 11 Oct 2023 10:37:41 AM UTC. 2023-10-11T10:41:01.937 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2023-10-11T10:41:01.938 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-10-11T10:41:01.939 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repository Size 2023-10-11T10:41:01.939 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-10-11T10:41:01.939 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2023-10-11T10:41:01.939 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-volume noarch 2:18.0.0-6661.g4bf94f59.el8 ceph-noarch 261 k 2023-10-11T10:41:01.939 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-11T10:41:01.940 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2023-10-11T10:41:01.940 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-10-11T10:41:01.940 INFO:teuthology.orchestra.run.smithi050.stdout:Install 1 Package 2023-10-11T10:41:01.940 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-11T10:41:01.940 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 261 k 2023-10-11T10:41:01.940 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 1.3 M 2023-10-11T10:41:01.941 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2023-10-11T10:41:02.264 INFO:teuthology.orchestra.run.smithi050.stdout:ceph-volume-18.0.0-6661.g4bf94f59.el8.noarch.rp 806 kB/s | 261 kB 00:00 2023-10-11T10:41:02.264 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2023-10-11T10:41:02.265 INFO:teuthology.orchestra.run.smithi050.stdout:Total 803 kB/s | 261 kB 00:00 2023-10-11T10:41:02.265 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2023-10-11T10:41:02.276 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2023-10-11T10:41:02.276 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2023-10-11T10:41:02.295 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2023-10-11T10:41:02.300 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2023-10-11T10:41:02.472 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2023-10-11T10:41:02.687 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-volume-2:18.0.0-6661.g4bf94f59.el8.noarch 1/1 2023-10-11T10:41:03.070 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-volume-2:18.0.0-6661.g4bf94f59.el8.noarch 1/1 2023-10-11T10:41:03.330 INFO:teuthology.orchestra.run.smithi111.stdout:Installed products updated. 2023-10-11T10:41:03.513 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T10:41:03.514 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2023-10-11T10:41:03.514 INFO:teuthology.orchestra.run.smithi111.stdout: librados-devel-2:18.0.0-6661.g4bf94f59.el8.x86_64 2023-10-11T10:41:03.514 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T10:41:03.514 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2023-10-11T10:41:03.677 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install libcephfs2 2023-10-11T10:41:04.033 INFO:teuthology.orchestra.run.smithi111.stdout:Updating Subscription Management repositories. 2023-10-11T10:41:04.162 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-volume-2:18.0.0-6661.g4bf94f59.el8.noarch 1/1 2023-10-11T10:41:04.162 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:03:23 ago on Wed 11 Oct 2023 10:37:41 AM UTC. 2023-10-11T10:41:05.637 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:03:26 ago on Wed 11 Oct 2023 10:37:39 AM UTC. 2023-10-11T10:41:06.974 INFO:teuthology.orchestra.run.smithi111.stdout:Package libcephfs2-2:18.0.0-6661.g4bf94f59.el8.x86_64 is already installed. 2023-10-11T10:41:07.082 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2023-10-11T10:41:07.084 INFO:teuthology.orchestra.run.smithi111.stdout:Nothing to do. 2023-10-11T10:41:07.084 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2023-10-11T10:41:07.227 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install libcephfs-devel 2023-10-11T10:41:07.577 INFO:teuthology.orchestra.run.smithi111.stdout:Updating Subscription Management repositories. 2023-10-11T10:41:09.079 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:03:30 ago on Wed 11 Oct 2023 10:37:39 AM UTC. 2023-10-11T10:41:09.507 INFO:teuthology.orchestra.run.smithi050.stdout:Installed products updated. 2023-10-11T10:41:09.709 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-11T10:41:09.709 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2023-10-11T10:41:09.710 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-volume-2:18.0.0-6661.g4bf94f59.el8.noarch 2023-10-11T10:41:09.710 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-11T10:41:09.710 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2023-10-11T10:41:09.874 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install librados-devel 2023-10-11T10:41:10.252 INFO:teuthology.orchestra.run.smithi050.stdout:Updating Subscription Management repositories. 2023-10-11T10:41:10.522 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2023-10-11T10:41:10.523 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2023-10-11T10:41:10.523 INFO:teuthology.orchestra.run.smithi111.stdout: Package Arch Version Repo Size 2023-10-11T10:41:10.523 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2023-10-11T10:41:10.524 INFO:teuthology.orchestra.run.smithi111.stdout:Installing: 2023-10-11T10:41:10.524 INFO:teuthology.orchestra.run.smithi111.stdout: libcephfs-devel x86_64 2:18.0.0-6661.g4bf94f59.el8 ceph 31 k 2023-10-11T10:41:10.524 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T10:41:10.524 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2023-10-11T10:41:10.525 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2023-10-11T10:41:10.525 INFO:teuthology.orchestra.run.smithi111.stdout:Install 1 Package 2023-10-11T10:41:10.525 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T10:41:10.525 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 31 k 2023-10-11T10:41:10.526 INFO:teuthology.orchestra.run.smithi111.stdout:Installed size: 139 k 2023-10-11T10:41:10.526 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2023-10-11T10:41:10.749 INFO:teuthology.orchestra.run.smithi111.stdout:libcephfs-devel-18.0.0-6661.g4bf94f59.el8.x86_6 138 kB/s | 31 kB 00:00 2023-10-11T10:41:10.749 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2023-10-11T10:41:10.749 INFO:teuthology.orchestra.run.smithi111.stdout:Total 137 kB/s | 31 kB 00:00 2023-10-11T10:41:10.749 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2023-10-11T10:41:10.752 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2023-10-11T10:41:10.753 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2023-10-11T10:41:10.762 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2023-10-11T10:41:10.766 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2023-10-11T10:41:10.819 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2023-10-11T10:41:11.020 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : libcephfs-devel-2:18.0.0-6661.g4bf94f59.el8.x86_64 1/1 2023-10-11T10:41:11.235 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: libcephfs-devel-2:18.0.0-6661.g4bf94f59.el8.x86_64 1/1 2023-10-11T10:41:11.800 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:03:30 ago on Wed 11 Oct 2023 10:37:41 AM UTC. 2023-10-11T10:41:12.228 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : libcephfs-devel-2:18.0.0-6661.g4bf94f59.el8.x86_64 1/1 2023-10-11T10:41:12.228 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:03:33 ago on Wed 11 Oct 2023 10:37:39 AM UTC. 2023-10-11T10:41:13.345 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2023-10-11T10:41:13.345 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-10-11T10:41:13.346 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repo Size 2023-10-11T10:41:13.346 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-10-11T10:41:13.346 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2023-10-11T10:41:13.346 INFO:teuthology.orchestra.run.smithi050.stdout: librados-devel x86_64 2:18.0.0-6661.g4bf94f59.el8 ceph 139 k 2023-10-11T10:41:13.347 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-11T10:41:13.347 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2023-10-11T10:41:13.347 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-10-11T10:41:13.347 INFO:teuthology.orchestra.run.smithi050.stdout:Install 1 Package 2023-10-11T10:41:13.347 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-11T10:41:13.348 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 139 k 2023-10-11T10:41:13.348 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 513 k 2023-10-11T10:41:13.348 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2023-10-11T10:41:13.895 INFO:teuthology.orchestra.run.smithi050.stdout:librados-devel-18.0.0-6661.g4bf94f59.el8.x86_64 253 kB/s | 139 kB 00:00 2023-10-11T10:41:13.896 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2023-10-11T10:41:13.897 INFO:teuthology.orchestra.run.smithi050.stdout:Total 252 kB/s | 139 kB 00:00 2023-10-11T10:41:13.897 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2023-10-11T10:41:13.903 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2023-10-11T10:41:13.903 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2023-10-11T10:41:13.966 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2023-10-11T10:41:13.970 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2023-10-11T10:41:14.091 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2023-10-11T10:41:14.250 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : librados-devel-2:18.0.0-6661.g4bf94f59.el8.x86_64 1/1 2023-10-11T10:41:14.491 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: librados-devel-2:18.0.0-6661.g4bf94f59.el8.x86_64 1/1 2023-10-11T10:41:15.551 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : librados-devel-2:18.0.0-6661.g4bf94f59.el8.x86_64 1/1 2023-10-11T10:41:15.551 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:03:34 ago on Wed 11 Oct 2023 10:37:41 AM UTC. 2023-10-11T10:41:16.678 INFO:teuthology.orchestra.run.smithi111.stdout:Installed products updated. 2023-10-11T10:41:16.858 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T10:41:16.859 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2023-10-11T10:41:16.859 INFO:teuthology.orchestra.run.smithi111.stdout: libcephfs-devel-2:18.0.0-6661.g4bf94f59.el8.x86_64 2023-10-11T10:41:16.859 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T10:41:16.859 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2023-10-11T10:41:16.990 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install librados2 2023-10-11T10:41:17.341 INFO:teuthology.orchestra.run.smithi111.stdout:Updating Subscription Management repositories. 2023-10-11T10:41:18.882 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:03:39 ago on Wed 11 Oct 2023 10:37:39 AM UTC. 2023-10-11T10:41:20.230 INFO:teuthology.orchestra.run.smithi111.stdout:Package librados2-2:18.0.0-6661.g4bf94f59.el8.x86_64 is already installed. 2023-10-11T10:41:20.339 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2023-10-11T10:41:20.339 INFO:teuthology.orchestra.run.smithi111.stdout:Nothing to do. 2023-10-11T10:41:20.340 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2023-10-11T10:41:20.443 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install librbd1 2023-10-11T10:41:20.795 INFO:teuthology.orchestra.run.smithi111.stdout:Updating Subscription Management repositories. 2023-10-11T10:41:21.227 INFO:teuthology.orchestra.run.smithi050.stdout:Installed products updated. 2023-10-11T10:41:21.429 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-11T10:41:21.429 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2023-10-11T10:41:21.429 INFO:teuthology.orchestra.run.smithi050.stdout: librados-devel-2:18.0.0-6661.g4bf94f59.el8.x86_64 2023-10-11T10:41:21.429 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-11T10:41:21.429 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2023-10-11T10:41:21.563 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install libcephfs2 2023-10-11T10:41:21.940 INFO:teuthology.orchestra.run.smithi050.stdout:Updating Subscription Management repositories. 2023-10-11T10:41:22.373 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:03:43 ago on Wed 11 Oct 2023 10:37:39 AM UTC. 2023-10-11T10:41:23.408 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:03:42 ago on Wed 11 Oct 2023 10:37:41 AM UTC. 2023-10-11T10:41:23.642 INFO:teuthology.orchestra.run.smithi111.stdout:Package librbd1-2:18.0.0-6661.g4bf94f59.el8.x86_64 is already installed. 2023-10-11T10:41:23.750 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2023-10-11T10:41:23.752 INFO:teuthology.orchestra.run.smithi111.stdout:Nothing to do. 2023-10-11T10:41:23.752 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2023-10-11T10:41:23.853 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install python3-rados 2023-10-11T10:41:24.205 INFO:teuthology.orchestra.run.smithi111.stdout:Updating Subscription Management repositories. 2023-10-11T10:41:24.837 INFO:teuthology.orchestra.run.smithi050.stdout:Package libcephfs2-2:18.0.0-6661.g4bf94f59.el8.x86_64 is already installed. 2023-10-11T10:41:24.952 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2023-10-11T10:41:24.954 INFO:teuthology.orchestra.run.smithi050.stdout:Nothing to do. 2023-10-11T10:41:24.954 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2023-10-11T10:41:25.084 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install libcephfs-devel 2023-10-11T10:41:25.457 INFO:teuthology.orchestra.run.smithi050.stdout:Updating Subscription Management repositories. 2023-10-11T10:41:25.634 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:03:46 ago on Wed 11 Oct 2023 10:37:39 AM UTC. 2023-10-11T10:41:26.911 INFO:teuthology.orchestra.run.smithi111.stdout:Package python3-rados-2:18.0.0-6661.g4bf94f59.el8.x86_64 is already installed. 2023-10-11T10:41:27.018 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2023-10-11T10:41:27.019 INFO:teuthology.orchestra.run.smithi111.stdout:Nothing to do. 2023-10-11T10:41:27.020 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2023-10-11T10:41:27.126 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install python3-rgw 2023-10-11T10:41:27.139 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:03:46 ago on Wed 11 Oct 2023 10:37:41 AM UTC. 2023-10-11T10:41:27.479 INFO:teuthology.orchestra.run.smithi111.stdout:Updating Subscription Management repositories. 2023-10-11T10:41:28.684 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2023-10-11T10:41:28.685 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-10-11T10:41:28.686 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repo Size 2023-10-11T10:41:28.686 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-10-11T10:41:28.686 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2023-10-11T10:41:28.687 INFO:teuthology.orchestra.run.smithi050.stdout: libcephfs-devel x86_64 2:18.0.0-6661.g4bf94f59.el8 ceph 31 k 2023-10-11T10:41:28.687 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-11T10:41:28.687 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2023-10-11T10:41:28.687 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-10-11T10:41:28.688 INFO:teuthology.orchestra.run.smithi050.stdout:Install 1 Package 2023-10-11T10:41:28.688 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-11T10:41:28.688 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 31 k 2023-10-11T10:41:28.688 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 139 k 2023-10-11T10:41:28.689 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2023-10-11T10:41:29.030 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:03:50 ago on Wed 11 Oct 2023 10:37:39 AM UTC. 2023-10-11T10:41:29.173 INFO:teuthology.orchestra.run.smithi050.stdout:libcephfs-devel-18.0.0-6661.g4bf94f59.el8.x86_6 64 kB/s | 31 kB 00:00 2023-10-11T10:41:29.174 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2023-10-11T10:41:29.174 INFO:teuthology.orchestra.run.smithi050.stdout:Total 63 kB/s | 31 kB 00:00 2023-10-11T10:41:29.175 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2023-10-11T10:41:29.178 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2023-10-11T10:41:29.179 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2023-10-11T10:41:29.190 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2023-10-11T10:41:29.194 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2023-10-11T10:41:29.253 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2023-10-11T10:41:29.440 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : libcephfs-devel-2:18.0.0-6661.g4bf94f59.el8.x86_64 1/1 2023-10-11T10:41:29.653 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: libcephfs-devel-2:18.0.0-6661.g4bf94f59.el8.x86_64 1/1 2023-10-11T10:41:30.311 INFO:teuthology.orchestra.run.smithi111.stdout:Package python3-rgw-2:18.0.0-6661.g4bf94f59.el8.x86_64 is already installed. 2023-10-11T10:41:30.419 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2023-10-11T10:41:30.420 INFO:teuthology.orchestra.run.smithi111.stdout:Nothing to do. 2023-10-11T10:41:30.420 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2023-10-11T10:41:30.536 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install python3-cephfs 2023-10-11T10:41:30.888 INFO:teuthology.orchestra.run.smithi111.stdout:Updating Subscription Management repositories. 2023-10-11T10:41:32.523 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:03:53 ago on Wed 11 Oct 2023 10:37:39 AM UTC. 2023-10-11T10:41:33.869 INFO:teuthology.orchestra.run.smithi111.stdout:Package python3-cephfs-2:18.0.0-6661.g4bf94f59.el8.x86_64 is already installed. 2023-10-11T10:41:33.978 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2023-10-11T10:41:33.979 INFO:teuthology.orchestra.run.smithi111.stdout:Nothing to do. 2023-10-11T10:41:33.979 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2023-10-11T10:41:34.085 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install python3-rbd 2023-10-11T10:41:34.437 INFO:teuthology.orchestra.run.smithi111.stdout:Updating Subscription Management repositories. 2023-10-11T10:41:36.004 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:03:57 ago on Wed 11 Oct 2023 10:37:39 AM UTC. 2023-10-11T10:41:37.269 INFO:teuthology.orchestra.run.smithi111.stdout:Package python3-rbd-2:18.0.0-6661.g4bf94f59.el8.x86_64 is already installed. 2023-10-11T10:41:37.376 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2023-10-11T10:41:37.377 INFO:teuthology.orchestra.run.smithi111.stdout:Nothing to do. 2023-10-11T10:41:37.377 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2023-10-11T10:41:37.483 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install rbd-fuse 2023-10-11T10:41:37.837 INFO:teuthology.orchestra.run.smithi111.stdout:Updating Subscription Management repositories. 2023-10-11T10:41:39.201 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:04:00 ago on Wed 11 Oct 2023 10:37:39 AM UTC. 2023-10-11T10:41:40.583 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2023-10-11T10:41:40.584 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2023-10-11T10:41:40.584 INFO:teuthology.orchestra.run.smithi111.stdout: Package Architecture Version Repository Size 2023-10-11T10:41:40.584 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2023-10-11T10:41:40.585 INFO:teuthology.orchestra.run.smithi111.stdout:Installing: 2023-10-11T10:41:40.585 INFO:teuthology.orchestra.run.smithi111.stdout: rbd-fuse x86_64 2:18.0.0-6661.g4bf94f59.el8 ceph 94 k 2023-10-11T10:41:40.585 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T10:41:40.586 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2023-10-11T10:41:40.586 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2023-10-11T10:41:40.586 INFO:teuthology.orchestra.run.smithi111.stdout:Install 1 Package 2023-10-11T10:41:40.586 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T10:41:40.587 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 94 k 2023-10-11T10:41:40.587 INFO:teuthology.orchestra.run.smithi111.stdout:Installed size: 276 k 2023-10-11T10:41:40.587 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2023-10-11T10:41:40.715 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : libcephfs-devel-2:18.0.0-6661.g4bf94f59.el8.x86_64 1/1 2023-10-11T10:41:40.715 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:03:59 ago on Wed 11 Oct 2023 10:37:41 AM UTC. 2023-10-11T10:41:40.849 INFO:teuthology.orchestra.run.smithi111.stdout:rbd-fuse-18.0.0-6661.g4bf94f59.el8.x86_64.rpm 353 kB/s | 94 kB 00:00 2023-10-11T10:41:40.850 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2023-10-11T10:41:40.850 INFO:teuthology.orchestra.run.smithi111.stdout:Total 351 kB/s | 94 kB 00:00 2023-10-11T10:41:40.851 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2023-10-11T10:41:40.861 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2023-10-11T10:41:40.861 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2023-10-11T10:41:40.914 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2023-10-11T10:41:40.917 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2023-10-11T10:41:41.023 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2023-10-11T10:41:41.206 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : rbd-fuse-2:18.0.0-6661.g4bf94f59.el8.x86_64 1/1 2023-10-11T10:41:41.435 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: rbd-fuse-2:18.0.0-6661.g4bf94f59.el8.x86_64 1/1 2023-10-11T10:41:42.387 INFO:teuthology.orchestra.run.smithi050.stdout:Installed products updated. 2023-10-11T10:41:42.508 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : rbd-fuse-2:18.0.0-6661.g4bf94f59.el8.x86_64 1/1 2023-10-11T10:41:42.509 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:04:03 ago on Wed 11 Oct 2023 10:37:39 AM UTC. 2023-10-11T10:41:42.587 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-11T10:41:42.587 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2023-10-11T10:41:42.588 INFO:teuthology.orchestra.run.smithi050.stdout: libcephfs-devel-2:18.0.0-6661.g4bf94f59.el8.x86_64 2023-10-11T10:41:42.588 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-11T10:41:42.588 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2023-10-11T10:41:42.732 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install librados2 2023-10-11T10:41:43.112 INFO:teuthology.orchestra.run.smithi050.stdout:Updating Subscription Management repositories. 2023-10-11T10:41:44.753 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:04:03 ago on Wed 11 Oct 2023 10:37:41 AM UTC. 2023-10-11T10:41:46.133 INFO:teuthology.orchestra.run.smithi050.stdout:Package librados2-2:18.0.0-6661.g4bf94f59.el8.x86_64 is already installed. 2023-10-11T10:41:46.249 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2023-10-11T10:41:46.251 INFO:teuthology.orchestra.run.smithi050.stdout:Nothing to do. 2023-10-11T10:41:46.251 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2023-10-11T10:41:46.367 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install librbd1 2023-10-11T10:41:46.742 INFO:teuthology.orchestra.run.smithi050.stdout:Updating Subscription Management repositories. 2023-10-11T10:41:47.581 INFO:teuthology.orchestra.run.smithi111.stdout:Installed products updated. 2023-10-11T10:41:47.766 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T10:41:47.766 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2023-10-11T10:41:47.766 INFO:teuthology.orchestra.run.smithi111.stdout: rbd-fuse-2:18.0.0-6661.g4bf94f59.el8.x86_64 2023-10-11T10:41:47.767 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T10:41:47.767 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2023-10-11T10:41:47.919 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install rbd-mirror 2023-10-11T10:41:48.171 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:04:07 ago on Wed 11 Oct 2023 10:37:41 AM UTC. 2023-10-11T10:41:48.273 INFO:teuthology.orchestra.run.smithi111.stdout:Updating Subscription Management repositories. 2023-10-11T10:41:49.600 INFO:teuthology.orchestra.run.smithi050.stdout:Package librbd1-2:18.0.0-6661.g4bf94f59.el8.x86_64 is already installed. 2023-10-11T10:41:49.722 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2023-10-11T10:41:49.723 INFO:teuthology.orchestra.run.smithi050.stdout:Nothing to do. 2023-10-11T10:41:49.723 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2023-10-11T10:41:49.766 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:04:10 ago on Wed 11 Oct 2023 10:37:39 AM UTC. 2023-10-11T10:41:49.844 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install python3-rados 2023-10-11T10:41:50.219 INFO:teuthology.orchestra.run.smithi050.stdout:Updating Subscription Management repositories. 2023-10-11T10:41:51.157 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2023-10-11T10:41:51.158 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2023-10-11T10:41:51.158 INFO:teuthology.orchestra.run.smithi111.stdout: Package Arch Version Repo Size 2023-10-11T10:41:51.158 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2023-10-11T10:41:51.159 INFO:teuthology.orchestra.run.smithi111.stdout:Installing: 2023-10-11T10:41:51.159 INFO:teuthology.orchestra.run.smithi111.stdout: rbd-mirror x86_64 2:18.0.0-6661.g4bf94f59.el8 ceph 3.6 M 2023-10-11T10:41:51.159 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T10:41:51.159 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2023-10-11T10:41:51.159 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2023-10-11T10:41:51.160 INFO:teuthology.orchestra.run.smithi111.stdout:Install 1 Package 2023-10-11T10:41:51.160 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T10:41:51.160 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 3.6 M 2023-10-11T10:41:51.160 INFO:teuthology.orchestra.run.smithi111.stdout:Installed size: 17 M 2023-10-11T10:41:51.160 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2023-10-11T10:41:51.568 INFO:teuthology.orchestra.run.smithi111.stdout:rbd-mirror-18.0.0-6661.g4bf94f59.el8.x86_64.rpm 8.8 MB/s | 3.6 MB 00:00 2023-10-11T10:41:51.568 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2023-10-11T10:41:51.569 INFO:teuthology.orchestra.run.smithi111.stdout:Total 8.7 MB/s | 3.6 MB 00:00 2023-10-11T10:41:51.569 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2023-10-11T10:41:51.581 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2023-10-11T10:41:51.582 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2023-10-11T10:41:51.648 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2023-10-11T10:41:51.652 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2023-10-11T10:41:51.801 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:04:10 ago on Wed 11 Oct 2023 10:37:41 AM UTC. 2023-10-11T10:41:52.107 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2023-10-11T10:41:52.324 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : rbd-mirror-2:18.0.0-6661.g4bf94f59.el8.x86_64 1/1 2023-10-11T10:41:52.659 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: rbd-mirror-2:18.0.0-6661.g4bf94f59.el8.x86_64 1/1 2023-10-11T10:41:53.238 INFO:teuthology.orchestra.run.smithi050.stdout:Package python3-rados-2:18.0.0-6661.g4bf94f59.el8.x86_64 is already installed. 2023-10-11T10:41:53.355 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2023-10-11T10:41:53.356 INFO:teuthology.orchestra.run.smithi050.stdout:Nothing to do. 2023-10-11T10:41:53.357 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2023-10-11T10:41:53.476 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install python3-rgw 2023-10-11T10:41:53.838 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : rbd-mirror-2:18.0.0-6661.g4bf94f59.el8.x86_64 1/1 2023-10-11T10:41:53.839 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:04:14 ago on Wed 11 Oct 2023 10:37:39 AM UTC. 2023-10-11T10:41:53.852 INFO:teuthology.orchestra.run.smithi050.stdout:Updating Subscription Management repositories. 2023-10-11T10:41:55.485 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:04:14 ago on Wed 11 Oct 2023 10:37:41 AM UTC. 2023-10-11T10:41:56.507 INFO:teuthology.orchestra.run.smithi111.stdout:Installed products updated. 2023-10-11T10:41:56.688 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T10:41:56.689 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2023-10-11T10:41:56.689 INFO:teuthology.orchestra.run.smithi111.stdout: rbd-mirror-2:18.0.0-6661.g4bf94f59.el8.x86_64 2023-10-11T10:41:56.689 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T10:41:56.689 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2023-10-11T10:41:56.832 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install rbd-nbd 2023-10-11T10:41:56.921 INFO:teuthology.orchestra.run.smithi050.stdout:Package python3-rgw-2:18.0.0-6661.g4bf94f59.el8.x86_64 is already installed. 2023-10-11T10:41:57.038 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2023-10-11T10:41:57.040 INFO:teuthology.orchestra.run.smithi050.stdout:Nothing to do. 2023-10-11T10:41:57.040 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2023-10-11T10:41:57.166 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install python3-cephfs 2023-10-11T10:41:57.186 INFO:teuthology.orchestra.run.smithi111.stdout:Updating Subscription Management repositories. 2023-10-11T10:41:57.540 INFO:teuthology.orchestra.run.smithi050.stdout:Updating Subscription Management repositories. 2023-10-11T10:41:58.024 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:04:19 ago on Wed 11 Oct 2023 10:37:39 AM UTC. 2023-10-11T10:41:58.922 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:04:17 ago on Wed 11 Oct 2023 10:37:41 AM UTC. 2023-10-11T10:41:59.477 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2023-10-11T10:41:59.478 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2023-10-11T10:41:59.478 INFO:teuthology.orchestra.run.smithi111.stdout: Package Architecture Version Repository Size 2023-10-11T10:41:59.479 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2023-10-11T10:41:59.479 INFO:teuthology.orchestra.run.smithi111.stdout:Installing: 2023-10-11T10:41:59.479 INFO:teuthology.orchestra.run.smithi111.stdout: rbd-nbd x86_64 2:18.0.0-6661.g4bf94f59.el8 ceph 179 k 2023-10-11T10:41:59.479 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T10:41:59.479 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2023-10-11T10:41:59.480 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2023-10-11T10:41:59.480 INFO:teuthology.orchestra.run.smithi111.stdout:Install 1 Package 2023-10-11T10:41:59.480 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T10:41:59.480 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 179 k 2023-10-11T10:41:59.480 INFO:teuthology.orchestra.run.smithi111.stdout:Installed size: 564 k 2023-10-11T10:41:59.481 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2023-10-11T10:41:59.772 INFO:teuthology.orchestra.run.smithi111.stdout:rbd-nbd-18.0.0-6661.g4bf94f59.el8.x86_64.rpm 611 kB/s | 179 kB 00:00 2023-10-11T10:41:59.772 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2023-10-11T10:41:59.772 INFO:teuthology.orchestra.run.smithi111.stdout:Total 609 kB/s | 179 kB 00:00 2023-10-11T10:41:59.773 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2023-10-11T10:41:59.784 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2023-10-11T10:41:59.784 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2023-10-11T10:41:59.837 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2023-10-11T10:41:59.841 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2023-10-11T10:41:59.962 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2023-10-11T10:42:00.146 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : rbd-nbd-2:18.0.0-6661.g4bf94f59.el8.x86_64 1/1 2023-10-11T10:42:00.367 INFO:teuthology.orchestra.run.smithi050.stdout:Package python3-cephfs-2:18.0.0-6661.g4bf94f59.el8.x86_64 is already installed. 2023-10-11T10:42:00.376 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: rbd-nbd-2:18.0.0-6661.g4bf94f59.el8.x86_64 1/1 2023-10-11T10:42:00.485 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2023-10-11T10:42:00.486 INFO:teuthology.orchestra.run.smithi050.stdout:Nothing to do. 2023-10-11T10:42:00.486 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2023-10-11T10:42:00.614 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install python3-rbd 2023-10-11T10:42:00.991 INFO:teuthology.orchestra.run.smithi050.stdout:Updating Subscription Management repositories. 2023-10-11T10:42:01.454 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : rbd-nbd-2:18.0.0-6661.g4bf94f59.el8.x86_64 1/1 2023-10-11T10:42:01.454 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:04:22 ago on Wed 11 Oct 2023 10:37:39 AM UTC. 2023-10-11T10:42:01.922 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:04:20 ago on Wed 11 Oct 2023 10:37:41 AM UTC. 2023-10-11T10:42:03.359 INFO:teuthology.orchestra.run.smithi050.stdout:Package python3-rbd-2:18.0.0-6661.g4bf94f59.el8.x86_64 is already installed. 2023-10-11T10:42:03.478 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2023-10-11T10:42:03.480 INFO:teuthology.orchestra.run.smithi050.stdout:Nothing to do. 2023-10-11T10:42:03.480 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2023-10-11T10:42:03.723 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install rbd-fuse 2023-10-11T10:42:03.957 INFO:teuthology.orchestra.run.smithi111.stdout:Installed products updated. 2023-10-11T10:42:04.097 INFO:teuthology.orchestra.run.smithi050.stdout:Updating Subscription Management repositories. 2023-10-11T10:42:04.139 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T10:42:04.139 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2023-10-11T10:42:04.139 INFO:teuthology.orchestra.run.smithi111.stdout: rbd-nbd-2:18.0.0-6661.g4bf94f59.el8.x86_64 2023-10-11T10:42:04.140 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T10:42:04.140 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2023-10-11T10:42:04.297 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install rbd-mirror 2023-10-11T10:42:04.652 INFO:teuthology.orchestra.run.smithi111.stdout:Updating Subscription Management repositories. 2023-10-11T10:42:04.939 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:04:23 ago on Wed 11 Oct 2023 10:37:41 AM UTC. 2023-10-11T10:42:05.438 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:04:26 ago on Wed 11 Oct 2023 10:37:39 AM UTC. 2023-10-11T10:42:06.482 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2023-10-11T10:42:06.483 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-10-11T10:42:06.483 INFO:teuthology.orchestra.run.smithi050.stdout: Package Architecture Version Repository Size 2023-10-11T10:42:06.484 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-10-11T10:42:06.484 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2023-10-11T10:42:06.484 INFO:teuthology.orchestra.run.smithi050.stdout: rbd-fuse x86_64 2:18.0.0-6661.g4bf94f59.el8 ceph 94 k 2023-10-11T10:42:06.484 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-11T10:42:06.484 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2023-10-11T10:42:06.485 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-10-11T10:42:06.485 INFO:teuthology.orchestra.run.smithi050.stdout:Install 1 Package 2023-10-11T10:42:06.485 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-11T10:42:06.485 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 94 k 2023-10-11T10:42:06.485 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 276 k 2023-10-11T10:42:06.485 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2023-10-11T10:42:06.679 INFO:teuthology.orchestra.run.smithi050.stdout:rbd-fuse-18.0.0-6661.g4bf94f59.el8.x86_64.rpm 481 kB/s | 94 kB 00:00 2023-10-11T10:42:06.679 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2023-10-11T10:42:06.680 INFO:teuthology.orchestra.run.smithi050.stdout:Total 478 kB/s | 94 kB 00:00 2023-10-11T10:42:06.680 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2023-10-11T10:42:06.692 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2023-10-11T10:42:06.692 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2023-10-11T10:42:06.705 INFO:teuthology.orchestra.run.smithi111.stdout:Package rbd-mirror-2:18.0.0-6661.g4bf94f59.el8.x86_64 is already installed. 2023-10-11T10:42:06.749 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2023-10-11T10:42:06.753 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2023-10-11T10:42:06.817 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2023-10-11T10:42:06.819 INFO:teuthology.orchestra.run.smithi111.stdout:Nothing to do. 2023-10-11T10:42:06.819 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2023-10-11T10:42:06.867 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2023-10-11T10:42:06.924 DEBUG:teuthology.parallel:result is None 2023-10-11T10:42:07.043 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : rbd-fuse-2:18.0.0-6661.g4bf94f59.el8.x86_64 1/1 2023-10-11T10:42:07.285 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: rbd-fuse-2:18.0.0-6661.g4bf94f59.el8.x86_64 1/1 2023-10-11T10:42:08.380 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : rbd-fuse-2:18.0.0-6661.g4bf94f59.el8.x86_64 1/1 2023-10-11T10:42:08.380 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:04:27 ago on Wed 11 Oct 2023 10:37:41 AM UTC. 2023-10-11T10:42:10.941 INFO:teuthology.orchestra.run.smithi050.stdout:Installed products updated. 2023-10-11T10:42:11.141 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-11T10:42:11.141 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2023-10-11T10:42:11.141 INFO:teuthology.orchestra.run.smithi050.stdout: rbd-fuse-2:18.0.0-6661.g4bf94f59.el8.x86_64 2023-10-11T10:42:11.142 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-11T10:42:11.142 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2023-10-11T10:42:11.288 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install rbd-mirror 2023-10-11T10:42:11.667 INFO:teuthology.orchestra.run.smithi050.stdout:Updating Subscription Management repositories. 2023-10-11T10:42:12.539 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:04:31 ago on Wed 11 Oct 2023 10:37:41 AM UTC. 2023-10-11T10:42:14.072 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2023-10-11T10:42:14.073 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-10-11T10:42:14.073 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repo Size 2023-10-11T10:42:14.073 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-10-11T10:42:14.074 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2023-10-11T10:42:14.074 INFO:teuthology.orchestra.run.smithi050.stdout: rbd-mirror x86_64 2:18.0.0-6661.g4bf94f59.el8 ceph 3.6 M 2023-10-11T10:42:14.074 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-11T10:42:14.074 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2023-10-11T10:42:14.075 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-10-11T10:42:14.075 INFO:teuthology.orchestra.run.smithi050.stdout:Install 1 Package 2023-10-11T10:42:14.075 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-11T10:42:14.075 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 3.6 M 2023-10-11T10:42:14.075 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 17 M 2023-10-11T10:42:14.076 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2023-10-11T10:42:15.091 INFO:teuthology.orchestra.run.smithi050.stdout:rbd-mirror-18.0.0-6661.g4bf94f59.el8.x86_64.rpm 3.5 MB/s | 3.6 MB 00:01 2023-10-11T10:42:15.092 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2023-10-11T10:42:15.093 INFO:teuthology.orchestra.run.smithi050.stdout:Total 3.5 MB/s | 3.6 MB 00:01 2023-10-11T10:42:15.093 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2023-10-11T10:42:15.108 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2023-10-11T10:42:15.108 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2023-10-11T10:42:15.184 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2023-10-11T10:42:15.188 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2023-10-11T10:42:15.705 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2023-10-11T10:42:15.886 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : rbd-mirror-2:18.0.0-6661.g4bf94f59.el8.x86_64 1/1 2023-10-11T10:42:16.257 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: rbd-mirror-2:18.0.0-6661.g4bf94f59.el8.x86_64 1/1 2023-10-11T10:42:17.312 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : rbd-mirror-2:18.0.0-6661.g4bf94f59.el8.x86_64 1/1 2023-10-11T10:42:17.312 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:04:36 ago on Wed 11 Oct 2023 10:37:41 AM UTC. 2023-10-11T10:42:19.817 INFO:teuthology.orchestra.run.smithi050.stdout:Installed products updated. 2023-10-11T10:42:20.016 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-11T10:42:20.016 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2023-10-11T10:42:20.017 INFO:teuthology.orchestra.run.smithi050.stdout: rbd-mirror-2:18.0.0-6661.g4bf94f59.el8.x86_64 2023-10-11T10:42:20.017 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-11T10:42:20.017 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2023-10-11T10:42:20.182 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install rbd-nbd 2023-10-11T10:42:20.560 INFO:teuthology.orchestra.run.smithi050.stdout:Updating Subscription Management repositories. 2023-10-11T10:42:21.377 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:04:40 ago on Wed 11 Oct 2023 10:37:41 AM UTC. 2023-10-11T10:42:22.923 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2023-10-11T10:42:22.924 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-10-11T10:42:22.924 INFO:teuthology.orchestra.run.smithi050.stdout: Package Architecture Version Repository Size 2023-10-11T10:42:22.924 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-10-11T10:42:22.925 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2023-10-11T10:42:22.925 INFO:teuthology.orchestra.run.smithi050.stdout: rbd-nbd x86_64 2:18.0.0-6661.g4bf94f59.el8 ceph 179 k 2023-10-11T10:42:22.925 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-11T10:42:22.925 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2023-10-11T10:42:22.925 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-10-11T10:42:22.925 INFO:teuthology.orchestra.run.smithi050.stdout:Install 1 Package 2023-10-11T10:42:22.926 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-11T10:42:22.926 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 179 k 2023-10-11T10:42:22.926 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 564 k 2023-10-11T10:42:22.926 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2023-10-11T10:42:23.212 INFO:teuthology.orchestra.run.smithi050.stdout:rbd-nbd-18.0.0-6661.g4bf94f59.el8.x86_64.rpm 623 kB/s | 179 kB 00:00 2023-10-11T10:42:23.212 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2023-10-11T10:42:23.213 INFO:teuthology.orchestra.run.smithi050.stdout:Total 620 kB/s | 179 kB 00:00 2023-10-11T10:42:23.213 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2023-10-11T10:42:23.226 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2023-10-11T10:42:23.226 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2023-10-11T10:42:23.285 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2023-10-11T10:42:23.289 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2023-10-11T10:42:23.408 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2023-10-11T10:42:23.558 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : rbd-nbd-2:18.0.0-6661.g4bf94f59.el8.x86_64 1/1 2023-10-11T10:42:23.792 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: rbd-nbd-2:18.0.0-6661.g4bf94f59.el8.x86_64 1/1 2023-10-11T10:42:24.844 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : rbd-nbd-2:18.0.0-6661.g4bf94f59.el8.x86_64 1/1 2023-10-11T10:42:24.845 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:04:43 ago on Wed 11 Oct 2023 10:37:41 AM UTC. 2023-10-11T10:42:27.499 INFO:teuthology.orchestra.run.smithi050.stdout:Installed products updated. 2023-10-11T10:42:27.693 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-11T10:42:27.694 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2023-10-11T10:42:27.694 INFO:teuthology.orchestra.run.smithi050.stdout: rbd-nbd-2:18.0.0-6661.g4bf94f59.el8.x86_64 2023-10-11T10:42:27.694 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-11T10:42:27.694 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2023-10-11T10:42:27.861 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install rbd-mirror 2023-10-11T10:42:28.245 INFO:teuthology.orchestra.run.smithi050.stdout:Updating Subscription Management repositories. 2023-10-11T10:42:29.102 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:04:48 ago on Wed 11 Oct 2023 10:37:41 AM UTC. 2023-10-11T10:42:30.534 INFO:teuthology.orchestra.run.smithi050.stdout:Package rbd-mirror-2:18.0.0-6661.g4bf94f59.el8.x86_64 is already installed. 2023-10-11T10:42:30.652 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2023-10-11T10:42:30.653 INFO:teuthology.orchestra.run.smithi050.stdout:Nothing to do. 2023-10-11T10:42:30.653 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2023-10-11T10:42:30.781 DEBUG:teuthology.parallel:result is None 2023-10-11T10:42:30.782 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=4bf94f59187ed7eba271559c0898cec750ad93d5 2023-10-11T10:42:31.000 DEBUG:teuthology.orchestra.run.smithi050:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-10-11T10:42:31.024 INFO:teuthology.orchestra.run.smithi050.stdout:18.0.0-6661.g4bf94f59.el8 2023-10-11T10:42:31.024 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-6661.g4bf94f59.el8 2023-10-11T10:42:31.024 INFO:teuthology.task.install:The correct ceph version 18.0.0-6661.g4bf94f59 is installed. 2023-10-11T10:42:31.025 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=4bf94f59187ed7eba271559c0898cec750ad93d5 2023-10-11T10:42:31.194 DEBUG:teuthology.orchestra.run.smithi111:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-10-11T10:42:31.223 INFO:teuthology.orchestra.run.smithi111.stdout:18.0.0-6661.g4bf94f59.el8 2023-10-11T10:42:31.223 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-6661.g4bf94f59.el8 2023-10-11T10:42:31.224 INFO:teuthology.task.install:The correct ceph version 18.0.0-6661.g4bf94f59 is installed. 2023-10-11T10:42:31.224 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-10-11T10:42:31.225 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2023-10-11T10:42:31.225 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-10-11T10:42:31.255 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2023-10-11T10:42:31.255 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-10-11T10:42:31.295 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-10-11T10:42:31.296 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2023-10-11T10:42:31.296 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd of=/usr/bin/daemon-helper 2023-10-11T10:42:31.325 DEBUG:teuthology.orchestra.run.smithi050:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-10-11T10:42:31.393 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2023-10-11T10:42:31.394 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd of=/usr/bin/daemon-helper 2023-10-11T10:42:31.425 DEBUG:teuthology.orchestra.run.smithi111:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-10-11T10:42:31.493 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-10-11T10:42:31.494 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2023-10-11T10:42:31.494 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd of=/usr/bin/adjust-ulimits 2023-10-11T10:42:31.523 DEBUG:teuthology.orchestra.run.smithi050:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-10-11T10:42:31.591 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2023-10-11T10:42:31.591 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd of=/usr/bin/adjust-ulimits 2023-10-11T10:42:31.619 DEBUG:teuthology.orchestra.run.smithi111:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-10-11T10:42:31.686 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-10-11T10:42:31.687 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2023-10-11T10:42:31.687 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd of=/usr/bin/stdin-killer 2023-10-11T10:42:31.715 DEBUG:teuthology.orchestra.run.smithi050:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-10-11T10:42:31.783 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2023-10-11T10:42:31.783 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd of=/usr/bin/stdin-killer 2023-10-11T10:42:31.811 DEBUG:teuthology.orchestra.run.smithi111:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-10-11T10:42:31.878 INFO:teuthology.run_tasks:Running task ceph... 2023-10-11T10:42:31.938 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2023-10-11T10:42:31.939 DEBUG:teuthology.orchestra.run.smithi050:> sudo chmod 777 /var/log/ceph 2023-10-11T10:42:31.941 DEBUG:teuthology.orchestra.run.smithi111:> sudo chmod 777 /var/log/ceph 2023-10-11T10:42:31.968 INFO:tasks.ceph:Disabling ceph logrotate... 2023-10-11T10:42:31.968 DEBUG:teuthology.orchestra.run.smithi050:> sudo rm -f -- /etc/logrotate.d/ceph 2023-10-11T10:42:32.009 DEBUG:teuthology.orchestra.run.smithi111:> sudo rm -f -- /etc/logrotate.d/ceph 2023-10-11T10:42:32.035 INFO:tasks.ceph:Creating extra log directories... 2023-10-11T10:42:32.036 DEBUG:teuthology.orchestra.run.smithi050:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-10-11T10:42:32.076 DEBUG:teuthology.orchestra.run.smithi111:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-10-11T10:42:32.114 INFO:tasks.ceph:Creating ceph cluster cluster1... 2023-10-11T10:42:32.115 INFO:tasks.ceph:config {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'none'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'cpu_profile': set(), 'cluster': 'cluster1', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-10-11T10:42:32.115 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/dis-2023-10-11_10:22:06-rbd-wip-dis-testing-distro-default-smithi/7421102', 'branch': 'wip-dis-testing', 'description': 'rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-stupid policy/none rbd-mirror/four-per-cluster supported-random-distro$/{rhel_8} workloads/rbd-mirror-snapshot-stress-workunit-fast-diff}', 'email': 'idryomov@gmail.com', 'first_in_suite': False, 'job_id': '7421102', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'smithi', 'meta': [{'desc': 'run two ceph clusters and install rbd-mirror'}, {'desc': 'configure the permissions for client.mirror'}, {'desc': '2 ceph clusters with 1 mon and 3 osds each'}, {'desc': 'run four rbd-mirror daemons per cluster'}, {'desc': 'run the rbd_mirror_stress.sh workunit to test the rbd-mirror daemon'}], 'name': 'dis-2023-10-11_10:22:06-rbd-wip-dis-testing-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 3, 'size': 30}}], 'os_type': 'rhel', 'os_version': '8.6', 'overrides': {'admin_socket': {'branch': 'wip-dis-testing'}, 'ceph': {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'none'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': '4bf94f59187ed7eba271559c0898cec750ad93d5'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '4bf94f59187ed7eba271559c0898cec750ad93d5'}}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-dis-testing-qa', 'sha1': '3d099a3859cc211cf8dd2acdaf3194ba0e96ea9c'}}, 'owner': 'scheduled_dis@teuthology', 'priority': 74, 'repo': 'https://github.com/ceph/ceph-ci', 'roles': [['cluster1.mon.a', 'cluster1.mgr.x', 'cluster2.mgr.x', 'cluster1.osd.0', 'cluster1.osd.1', 'cluster1.osd.2', 'cluster1.client.0', 'cluster2.client.0'], ['cluster2.mon.a', 'cluster2.osd.0', 'cluster2.osd.1', 'cluster2.osd.2', 'cluster1.client.mirror', 'cluster1.client.mirror.0', 'cluster1.client.mirror.1', 'cluster1.client.mirror.2', 'cluster1.client.mirror.3', 'cluster1.client.mirror.4', 'cluster1.client.mirror.5', 'cluster1.client.mirror.6', 'cluster2.client.mirror', 'cluster2.client.mirror.0', 'cluster2.client.mirror.1', 'cluster2.client.mirror.2', 'cluster2.client.mirror.3', 'cluster2.client.mirror.4', 'cluster2.client.mirror.5', 'cluster2.client.mirror.6']], 'seed': 2449, 'sha1': '4bf94f59187ed7eba271559c0898cec750ad93d5', 'sleep_before_teardown': 0, 'subset': '49/99', 'suite': 'rbd', 'suite_branch': 'wip-dis-testing-qa', 'suite_path': '/home/teuthworker/src/github.com_idryomov_ceph_3d099a3859cc211cf8dd2acdaf3194ba0e96ea9c/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/idryomov/ceph', 'suite_sha1': '3d099a3859cc211cf8dd2acdaf3194ba0e96ea9c', 'targets': {'smithi050.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs=', 'smithi111.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': ['rbd-mirror'], 'flavor': 'default', 'sha1': '4bf94f59187ed7eba271559c0898cec750ad93d5'}}, {'ceph': {'cluster': 'cluster1', 'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'none'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': '4bf94f59187ed7eba271559c0898cec750ad93d5'}}, {'ceph': {'cluster': 'cluster2'}}, {'exec': {'cluster1.client.mirror.0': ["sudo ceph --cluster cluster1 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'"]}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.0', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.1', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.2', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.3', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.0', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.1', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.2', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.3', 'thrash': True}}, {'rbd-mirror-thrash': {'cluster': 'cluster1'}}, {'rbd-mirror-thrash': {'cluster': 'cluster2'}}, {'workunit': {'clients': {'cluster1.client.mirror': ['rbd/rbd_mirror_stress.sh']}, 'env': {'CEPH_ARGS': '', 'MIRROR_IMAGE_MODE': 'snapshot', 'MIRROR_POOL_MODE': 'image', 'RBD_IMAGE_FEATURES': 'layering,exclusive-lock,object-map,fast-diff', 'RBD_MIRROR_INSTANCES': '4', 'RBD_MIRROR_USE_EXISTING_CLUSTER': '1', 'RBD_MIRROR_USE_RBD_MIRROR': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '54e62bcbac4e53d9685e08328b790d3b20d71cae', 'timestamp': '2023-10-11_10:22:06', 'tube': 'smithi', 'user': 'dis', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2830'} 2023-10-11T10:42:32.116 DEBUG:teuthology.orchestra.run.smithi050:> install -d -m0755 -- /home/ubuntu/cephtest/cluster1.data 2023-10-11T10:42:32.144 DEBUG:teuthology.orchestra.run.smithi111:> install -d -m0755 -- /home/ubuntu/cephtest/cluster1.data 2023-10-11T10:42:32.160 DEBUG:teuthology.orchestra.run.smithi050:> sudo install -d -m0777 -- /var/run/ceph 2023-10-11T10:42:32.201 DEBUG:teuthology.orchestra.run.smithi111:> sudo install -d -m0777 -- /var/run/ceph 2023-10-11T10:42:32.227 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2023-10-11T10:42:32.228 DEBUG:teuthology.orchestra.run.smithi050:> dd if=/scratch_devs of=/dev/stdout 2023-10-11T10:42:32.284 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-10-11T10:42:32.284 DEBUG:teuthology.orchestra.run.smithi050:> stat /dev/vg_nvme/lv_1 2023-10-11T10:42:32.341 INFO:teuthology.orchestra.run.smithi050.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-10-11T10:42:32.341 INFO:teuthology.orchestra.run.smithi050.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-11T10:42:32.341 INFO:teuthology.orchestra.run.smithi050.stdout:Device: 6h/6d Inode: 389037 Links: 1 2023-10-11T10:42:32.342 INFO:teuthology.orchestra.run.smithi050.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-11T10:42:32.342 INFO:teuthology.orchestra.run.smithi050.stdout:Context: system_u:object_r:device_t:s0 2023-10-11T10:42:32.342 INFO:teuthology.orchestra.run.smithi050.stdout:Access: 2023-10-11 10:42:16.055029363 +0000 2023-10-11T10:42:32.342 INFO:teuthology.orchestra.run.smithi050.stdout:Modify: 2023-10-11 10:36:55.376541277 +0000 2023-10-11T10:42:32.342 INFO:teuthology.orchestra.run.smithi050.stdout:Change: 2023-10-11 10:36:55.376541277 +0000 2023-10-11T10:42:32.342 INFO:teuthology.orchestra.run.smithi050.stdout: Birth: - 2023-10-11T10:42:32.343 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-10-11T10:42:32.406 INFO:teuthology.orchestra.run.smithi050.stderr:1+0 records in 2023-10-11T10:42:32.406 INFO:teuthology.orchestra.run.smithi050.stderr:1+0 records out 2023-10-11T10:42:32.406 INFO:teuthology.orchestra.run.smithi050.stderr:512 bytes copied, 0.000113249 s, 4.5 MB/s 2023-10-11T10:42:32.407 DEBUG:teuthology.orchestra.run.smithi050:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-10-11T10:42:32.465 DEBUG:teuthology.orchestra.run.smithi050:> stat /dev/vg_nvme/lv_2 2023-10-11T10:42:32.522 INFO:teuthology.orchestra.run.smithi050.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-10-11T10:42:32.523 INFO:teuthology.orchestra.run.smithi050.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-11T10:42:32.523 INFO:teuthology.orchestra.run.smithi050.stdout:Device: 6h/6d Inode: 389086 Links: 1 2023-10-11T10:42:32.523 INFO:teuthology.orchestra.run.smithi050.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-11T10:42:32.523 INFO:teuthology.orchestra.run.smithi050.stdout:Context: system_u:object_r:device_t:s0 2023-10-11T10:42:32.524 INFO:teuthology.orchestra.run.smithi050.stdout:Access: 2023-10-11 10:42:16.055029363 +0000 2023-10-11T10:42:32.524 INFO:teuthology.orchestra.run.smithi050.stdout:Modify: 2023-10-11 10:36:55.863529031 +0000 2023-10-11T10:42:32.524 INFO:teuthology.orchestra.run.smithi050.stdout:Change: 2023-10-11 10:36:55.863529031 +0000 2023-10-11T10:42:32.524 INFO:teuthology.orchestra.run.smithi050.stdout: Birth: - 2023-10-11T10:42:32.524 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-10-11T10:42:32.588 INFO:teuthology.orchestra.run.smithi050.stderr:1+0 records in 2023-10-11T10:42:32.588 INFO:teuthology.orchestra.run.smithi050.stderr:1+0 records out 2023-10-11T10:42:32.588 INFO:teuthology.orchestra.run.smithi050.stderr:512 bytes copied, 0.000108275 s, 4.7 MB/s 2023-10-11T10:42:32.589 DEBUG:teuthology.orchestra.run.smithi050:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-10-11T10:42:32.647 DEBUG:teuthology.orchestra.run.smithi050:> stat /dev/vg_nvme/lv_3 2023-10-11T10:42:32.704 INFO:teuthology.orchestra.run.smithi050.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-10-11T10:42:32.704 INFO:teuthology.orchestra.run.smithi050.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-11T10:42:32.704 INFO:teuthology.orchestra.run.smithi050.stdout:Device: 6h/6d Inode: 387610 Links: 1 2023-10-11T10:42:32.704 INFO:teuthology.orchestra.run.smithi050.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-11T10:42:32.704 INFO:teuthology.orchestra.run.smithi050.stdout:Context: system_u:object_r:device_t:s0 2023-10-11T10:42:32.705 INFO:teuthology.orchestra.run.smithi050.stdout:Access: 2023-10-11 10:42:16.055029363 +0000 2023-10-11T10:42:32.705 INFO:teuthology.orchestra.run.smithi050.stdout:Modify: 2023-10-11 10:36:56.358516584 +0000 2023-10-11T10:42:32.705 INFO:teuthology.orchestra.run.smithi050.stdout:Change: 2023-10-11 10:36:56.358516584 +0000 2023-10-11T10:42:32.705 INFO:teuthology.orchestra.run.smithi050.stdout: Birth: - 2023-10-11T10:42:32.705 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-10-11T10:42:32.768 INFO:teuthology.orchestra.run.smithi050.stderr:1+0 records in 2023-10-11T10:42:32.769 INFO:teuthology.orchestra.run.smithi050.stderr:1+0 records out 2023-10-11T10:42:32.769 INFO:teuthology.orchestra.run.smithi050.stderr:512 bytes copied, 9.6516e-05 s, 5.3 MB/s 2023-10-11T10:42:32.770 DEBUG:teuthology.orchestra.run.smithi050:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-10-11T10:42:32.827 DEBUG:teuthology.orchestra.run.smithi050:> stat /dev/vg_nvme/lv_4 2023-10-11T10:42:32.883 INFO:teuthology.orchestra.run.smithi050.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-10-11T10:42:32.884 INFO:teuthology.orchestra.run.smithi050.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-11T10:42:32.884 INFO:teuthology.orchestra.run.smithi050.stdout:Device: 6h/6d Inode: 387030 Links: 1 2023-10-11T10:42:32.884 INFO:teuthology.orchestra.run.smithi050.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-11T10:42:32.884 INFO:teuthology.orchestra.run.smithi050.stdout:Context: system_u:object_r:device_t:s0 2023-10-11T10:42:32.884 INFO:teuthology.orchestra.run.smithi050.stdout:Access: 2023-10-11 10:42:16.055029363 +0000 2023-10-11T10:42:32.884 INFO:teuthology.orchestra.run.smithi050.stdout:Modify: 2023-10-11 10:36:56.859503987 +0000 2023-10-11T10:42:32.885 INFO:teuthology.orchestra.run.smithi050.stdout:Change: 2023-10-11 10:36:56.859503987 +0000 2023-10-11T10:42:32.885 INFO:teuthology.orchestra.run.smithi050.stdout: Birth: - 2023-10-11T10:42:32.885 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-10-11T10:42:32.949 INFO:teuthology.orchestra.run.smithi050.stderr:1+0 records in 2023-10-11T10:42:32.949 INFO:teuthology.orchestra.run.smithi050.stderr:1+0 records out 2023-10-11T10:42:32.950 INFO:teuthology.orchestra.run.smithi050.stderr:512 bytes copied, 0.0001142 s, 4.5 MB/s 2023-10-11T10:42:32.951 DEBUG:teuthology.orchestra.run.smithi050:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-10-11T10:42:33.008 INFO:tasks.ceph:osd dev map: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2023-10-11T10:42:33.008 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi050.front.sepia.ceph.com'): {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'}} 2023-10-11T10:42:33.009 INFO:tasks.ceph:Generating config... 2023-10-11T10:42:33.010 INFO:tasks.ceph:[client] debug rbd = 20 2023-10-11T10:42:33.010 INFO:tasks.ceph:[client] debug rbd_mirror = 15 2023-10-11T10:42:33.010 INFO:tasks.ceph:[client] log to stderr = False 2023-10-11T10:42:33.011 INFO:tasks.ceph:[client] rbd default features = 125 2023-10-11T10:42:33.011 INFO:tasks.ceph:[client] rbd mirror image policy type = none 2023-10-11T10:42:33.011 INFO:tasks.ceph:[client.mirror.0] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-10-11T10:42:33.011 INFO:tasks.ceph:[client.mirror.0] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-10-11T10:42:33.012 INFO:tasks.ceph:[client.mirror.1] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-10-11T10:42:33.012 INFO:tasks.ceph:[client.mirror.1] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-10-11T10:42:33.012 INFO:tasks.ceph:[client.mirror.2] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-10-11T10:42:33.012 INFO:tasks.ceph:[client.mirror.2] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-10-11T10:42:33.013 INFO:tasks.ceph:[client.mirror.3] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-10-11T10:42:33.013 INFO:tasks.ceph:[client.mirror.3] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-10-11T10:42:33.013 INFO:tasks.ceph:[global] mon client directed command retry = 5 2023-10-11T10:42:33.013 INFO:tasks.ceph:[global] mon warn on pool no app = False 2023-10-11T10:42:33.013 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2023-10-11T10:42:33.014 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-10-11T10:42:33.014 INFO:tasks.ceph:[mgr] debug ms = 1 2023-10-11T10:42:33.014 INFO:tasks.ceph:[mon] debug mon = 20 2023-10-11T10:42:33.015 INFO:tasks.ceph:[mon] debug ms = 1 2023-10-11T10:42:33.015 INFO:tasks.ceph:[mon] debug paxos = 20 2023-10-11T10:42:33.015 INFO:tasks.ceph:[osd] bdev async discard = True 2023-10-11T10:42:33.015 INFO:tasks.ceph:[osd] bdev enable discard = True 2023-10-11T10:42:33.016 INFO:tasks.ceph:[osd] bluestore allocator = stupid 2023-10-11T10:42:33.016 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2023-10-11T10:42:33.016 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2023-10-11T10:42:33.016 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2023-10-11T10:42:33.016 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2023-10-11T10:42:33.017 INFO:tasks.ceph:[osd] debug ms = 1 2023-10-11T10:42:33.017 INFO:tasks.ceph:[osd] debug osd = 20 2023-10-11T10:42:33.017 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2023-10-11T10:42:33.017 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2023-10-11T10:42:33.018 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2023-10-11T10:42:33.018 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2023-10-11T10:42:33.018 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2023-10-11T10:42:33.018 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2023-10-11T10:42:33.019 INFO:tasks.ceph:Setting up cluster1.mon.a... 2023-10-11T10:42:33.019 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/cluster1.keyring 2023-10-11T10:42:33.096 INFO:teuthology.orchestra.run.smithi050.stdout:creating /etc/ceph/cluster1.keyring 2023-10-11T10:42:33.099 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/cluster1.keyring 2023-10-11T10:42:33.145 DEBUG:teuthology.orchestra.run.smithi050:> sudo chmod 0644 /etc/ceph/cluster1.keyring 2023-10-11T10:42:33.171 DEBUG:tasks.ceph:Ceph mon addresses: [('cluster1.mon.a', '172.21.15.50')] 2023-10-11T10:42:33.172 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.50', 'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'none'}, 'mon.a': {}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}} 2023-10-11T10:42:33.173 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2023-10-11T10:42:33.173 DEBUG:teuthology.orchestra.run.smithi050:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-10-11T10:42:33.228 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.50 --print /home/ubuntu/cephtest/cluster1.monmap 2023-10-11T10:42:33.312 INFO:teuthology.orchestra.run.smithi050.stdout:monmaptool: monmap file /home/ubuntu/cephtest/cluster1.monmap 2023-10-11T10:42:33.312 INFO:teuthology.orchestra.run.smithi050.stdout:monmaptool: generated fsid 62c62aca-a3d3-4cfb-94c6-84fa1a529031 2023-10-11T10:42:33.312 INFO:teuthology.orchestra.run.smithi050.stdout:setting min_mon_release = pacific 2023-10-11T10:42:33.313 INFO:teuthology.orchestra.run.smithi050.stdout:epoch 0 2023-10-11T10:42:33.313 INFO:teuthology.orchestra.run.smithi050.stdout:fsid 62c62aca-a3d3-4cfb-94c6-84fa1a529031 2023-10-11T10:42:33.313 INFO:teuthology.orchestra.run.smithi050.stdout:last_changed 2023-10-11T10:42:33.312758+0000 2023-10-11T10:42:33.313 INFO:teuthology.orchestra.run.smithi050.stdout:created 2023-10-11T10:42:33.312758+0000 2023-10-11T10:42:33.313 INFO:teuthology.orchestra.run.smithi050.stdout:min_mon_release 16 (pacific) 2023-10-11T10:42:33.313 INFO:teuthology.orchestra.run.smithi050.stdout:election_strategy: 1 2023-10-11T10:42:33.314 INFO:teuthology.orchestra.run.smithi050.stdout:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-11T10:42:33.314 INFO:teuthology.orchestra.run.smithi050.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/cluster1.monmap (1 monitors) 2023-10-11T10:42:33.315 DEBUG:teuthology.orchestra.run.smithi050:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-10-11T10:42:33.331 INFO:tasks.ceph:Writing /etc/ceph/cluster1.conf for FSID 62c62aca-a3d3-4cfb-94c6-84fa1a529031... 2023-10-11T10:42:33.332 DEBUG:teuthology.orchestra.run.smithi050:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster1.conf && sudo chmod 0644 /etc/ceph/cluster1.conf > /dev/null 2023-10-11T10:42:33.374 DEBUG:teuthology.orchestra.run.smithi111:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster1.conf && sudo chmod 0644 /etc/ceph/cluster1.conf > /dev/null 2023-10-11T10:42:33.418 INFO:teuthology.orchestra.run.smithi050.stdout:[global] 2023-10-11T10:42:33.419 INFO:teuthology.orchestra.run.smithi050.stdout: chdir = "" 2023-10-11T10:42:33.419 INFO:teuthology.orchestra.run.smithi050.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-10-11T10:42:33.419 INFO:teuthology.orchestra.run.smithi050.stdout: auth supported = cephx 2023-10-11T10:42:33.419 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-11T10:42:33.419 INFO:teuthology.orchestra.run.smithi050.stdout: filestore xattr use omap = true 2023-10-11T10:42:33.420 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-11T10:42:33.420 INFO:teuthology.orchestra.run.smithi050.stdout: mon clock drift allowed = 1.000 2023-10-11T10:42:33.420 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-11T10:42:33.420 INFO:teuthology.orchestra.run.smithi050.stdout: osd crush chooseleaf type = 0 2023-10-11T10:42:33.420 INFO:teuthology.orchestra.run.smithi050.stdout: auth debug = true 2023-10-11T10:42:33.421 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-11T10:42:33.421 INFO:teuthology.orchestra.run.smithi050.stdout: ms die on old message = true 2023-10-11T10:42:33.421 INFO:teuthology.orchestra.run.smithi050.stdout: ms die on bug = true 2023-10-11T10:42:33.421 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-11T10:42:33.421 INFO:teuthology.orchestra.run.smithi050.stdout: mon max pg per osd = 10000 # >= luminous 2023-10-11T10:42:33.422 INFO:teuthology.orchestra.run.smithi050.stdout: mon pg warn max object skew = 0 2023-10-11T10:42:33.422 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-11T10:42:33.422 INFO:teuthology.orchestra.run.smithi050.stdout: # disable pg_autoscaler by default for new pools 2023-10-11T10:42:33.422 INFO:teuthology.orchestra.run.smithi050.stdout: osd_pool_default_pg_autoscale_mode = off 2023-10-11T10:42:33.422 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-11T10:42:33.422 INFO:teuthology.orchestra.run.smithi050.stdout: osd pool default size = 2 2023-10-11T10:42:33.423 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-11T10:42:33.423 INFO:teuthology.orchestra.run.smithi050.stdout: mon osd allow primary affinity = true 2023-10-11T10:42:33.423 INFO:teuthology.orchestra.run.smithi050.stdout: mon osd allow pg remap = true 2023-10-11T10:42:33.423 INFO:teuthology.orchestra.run.smithi050.stdout: mon warn on legacy crush tunables = false 2023-10-11T10:42:33.423 INFO:teuthology.orchestra.run.smithi050.stdout: mon warn on crush straw calc version zero = false 2023-10-11T10:42:33.424 INFO:teuthology.orchestra.run.smithi050.stdout: mon warn on no sortbitwise = false 2023-10-11T10:42:33.424 INFO:teuthology.orchestra.run.smithi050.stdout: mon warn on osd down out interval zero = false 2023-10-11T10:42:33.424 INFO:teuthology.orchestra.run.smithi050.stdout: mon warn on too few osds = false 2023-10-11T10:42:33.424 INFO:teuthology.orchestra.run.smithi050.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-10-11T10:42:33.424 INFO:teuthology.orchestra.run.smithi050.stdout: mon_warn_on_pool_no_redundancy = false 2023-10-11T10:42:33.424 INFO:teuthology.orchestra.run.smithi050.stdout: mon_allow_pool_size_one = true 2023-10-11T10:42:33.425 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-11T10:42:33.425 INFO:teuthology.orchestra.run.smithi050.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-10-11T10:42:33.425 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-11T10:42:33.425 INFO:teuthology.orchestra.run.smithi050.stdout: osd default data pool replay window = 5 2023-10-11T10:42:33.425 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-11T10:42:33.425 INFO:teuthology.orchestra.run.smithi050.stdout: mon allow pool delete = true 2023-10-11T10:42:33.426 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-11T10:42:33.426 INFO:teuthology.orchestra.run.smithi050.stdout: mon cluster log file level = debug 2023-10-11T10:42:33.426 INFO:teuthology.orchestra.run.smithi050.stdout: debug asserts on shutdown = true 2023-10-11T10:42:33.426 INFO:teuthology.orchestra.run.smithi050.stdout: mon health detail to clog = false 2023-10-11T10:42:33.426 INFO:teuthology.orchestra.run.smithi050.stdout: mon host = 172.21.15.50 2023-10-11T10:42:33.426 INFO:teuthology.orchestra.run.smithi050.stdout: mon client directed command retry = 5 2023-10-11T10:42:33.427 INFO:teuthology.orchestra.run.smithi050.stdout: mon warn on pool no app = False 2023-10-11T10:42:33.427 INFO:teuthology.orchestra.run.smithi050.stdout: ms inject socket failures = 5000 2023-10-11T10:42:33.427 INFO:teuthology.orchestra.run.smithi050.stdout: fsid = 62c62aca-a3d3-4cfb-94c6-84fa1a529031 2023-10-11T10:42:33.427 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-11T10:42:33.427 INFO:teuthology.orchestra.run.smithi050.stdout:[osd] 2023-10-11T10:42:33.428 INFO:teuthology.orchestra.run.smithi050.stdout: osd journal size = 100 2023-10-11T10:42:33.428 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-11T10:42:33.428 INFO:teuthology.orchestra.run.smithi050.stdout: osd scrub load threshold = 5.0 2023-10-11T10:42:33.428 INFO:teuthology.orchestra.run.smithi050.stdout: osd scrub max interval = 600 2023-10-11T10:42:33.428 INFO:teuthology.orchestra.run.smithi050.stdout: osd mclock profile = high_recovery_ops 2023-10-11T10:42:33.428 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-11T10:42:33.429 INFO:teuthology.orchestra.run.smithi050.stdout: osd recover clone overlap = true 2023-10-11T10:42:33.429 INFO:teuthology.orchestra.run.smithi050.stdout: osd recovery max chunk = 1048576 2023-10-11T10:42:33.429 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-11T10:42:33.429 INFO:teuthology.orchestra.run.smithi050.stdout: osd debug shutdown = true 2023-10-11T10:42:33.429 INFO:teuthology.orchestra.run.smithi050.stdout: osd debug op order = true 2023-10-11T10:42:33.430 INFO:teuthology.orchestra.run.smithi050.stdout: osd debug verify stray on activate = true 2023-10-11T10:42:33.430 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-11T10:42:33.430 INFO:teuthology.orchestra.run.smithi050.stdout: osd open classes on start = true 2023-10-11T10:42:33.430 INFO:teuthology.orchestra.run.smithi050.stdout: osd debug pg log writeout = true 2023-10-11T10:42:33.430 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-11T10:42:33.431 INFO:teuthology.orchestra.run.smithi050.stdout: osd deep scrub update digest min age = 30 2023-10-11T10:42:33.431 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-11T10:42:33.431 INFO:teuthology.orchestra.run.smithi050.stdout: osd map max advance = 10 2023-10-11T10:42:33.431 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-11T10:42:33.431 INFO:teuthology.orchestra.run.smithi050.stdout: journal zero on create = true 2023-10-11T10:42:33.431 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-11T10:42:33.432 INFO:teuthology.orchestra.run.smithi050.stdout: filestore ondisk finisher threads = 3 2023-10-11T10:42:33.432 INFO:teuthology.orchestra.run.smithi050.stdout: filestore apply finisher threads = 3 2023-10-11T10:42:33.432 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-11T10:42:33.432 INFO:teuthology.orchestra.run.smithi050.stdout: bdev debug aio = true 2023-10-11T10:42:33.432 INFO:teuthology.orchestra.run.smithi050.stdout: osd debug misdirected ops = true 2023-10-11T10:42:33.433 INFO:teuthology.orchestra.run.smithi050.stdout: bdev async discard = True 2023-10-11T10:42:33.433 INFO:teuthology.orchestra.run.smithi050.stdout: bdev enable discard = True 2023-10-11T10:42:33.433 INFO:teuthology.orchestra.run.smithi050.stdout: bluestore allocator = stupid 2023-10-11T10:42:33.433 INFO:teuthology.orchestra.run.smithi050.stdout: bluestore block size = 96636764160 2023-10-11T10:42:33.433 INFO:teuthology.orchestra.run.smithi050.stdout: bluestore fsck on mount = True 2023-10-11T10:42:33.434 INFO:teuthology.orchestra.run.smithi050.stdout: debug bluefs = 1/20 2023-10-11T10:42:33.434 INFO:teuthology.orchestra.run.smithi050.stdout: debug bluestore = 1/20 2023-10-11T10:42:33.434 INFO:teuthology.orchestra.run.smithi050.stdout: debug ms = 1 2023-10-11T10:42:33.434 INFO:teuthology.orchestra.run.smithi050.stdout: debug osd = 20 2023-10-11T10:42:33.434 INFO:teuthology.orchestra.run.smithi050.stdout: debug rocksdb = 4/10 2023-10-11T10:42:33.434 INFO:teuthology.orchestra.run.smithi050.stdout: mon osd backfillfull_ratio = 0.85 2023-10-11T10:42:33.435 INFO:teuthology.orchestra.run.smithi050.stdout: mon osd full ratio = 0.9 2023-10-11T10:42:33.435 INFO:teuthology.orchestra.run.smithi050.stdout: mon osd nearfull ratio = 0.8 2023-10-11T10:42:33.435 INFO:teuthology.orchestra.run.smithi050.stdout: osd failsafe full ratio = 0.95 2023-10-11T10:42:33.435 INFO:teuthology.orchestra.run.smithi050.stdout: osd objectstore = bluestore 2023-10-11T10:42:33.435 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-11T10:42:33.435 INFO:teuthology.orchestra.run.smithi050.stdout:[mgr] 2023-10-11T10:42:33.436 INFO:teuthology.orchestra.run.smithi050.stdout: debug ms = 1 2023-10-11T10:42:33.436 INFO:teuthology.orchestra.run.smithi050.stdout: debug mgr = 20 2023-10-11T10:42:33.436 INFO:teuthology.orchestra.run.smithi050.stdout: debug mon = 20 2023-10-11T10:42:33.436 INFO:teuthology.orchestra.run.smithi050.stdout: debug auth = 20 2023-10-11T10:42:33.436 INFO:teuthology.orchestra.run.smithi050.stdout: mon reweight min pgs per osd = 4 2023-10-11T10:42:33.436 INFO:teuthology.orchestra.run.smithi050.stdout: mon reweight min bytes per osd = 10 2023-10-11T10:42:33.437 INFO:teuthology.orchestra.run.smithi050.stdout: mgr/telemetry/nag = false 2023-10-11T10:42:33.437 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-11T10:42:33.437 INFO:teuthology.orchestra.run.smithi050.stdout:[mon] 2023-10-11T10:42:33.437 INFO:teuthology.orchestra.run.smithi050.stdout: debug ms = 1 2023-10-11T10:42:33.437 INFO:teuthology.orchestra.run.smithi050.stdout: debug mon = 20 2023-10-11T10:42:33.438 INFO:teuthology.orchestra.run.smithi050.stdout: debug paxos = 20 2023-10-11T10:42:33.438 INFO:teuthology.orchestra.run.smithi050.stdout: debug auth = 20 2023-10-11T10:42:33.438 INFO:teuthology.orchestra.run.smithi050.stdout: mon data avail warn = 5 2023-10-11T10:42:33.438 INFO:teuthology.orchestra.run.smithi050.stdout: mon mgr mkfs grace = 240 2023-10-11T10:42:33.438 INFO:teuthology.orchestra.run.smithi050.stdout: mon reweight min pgs per osd = 4 2023-10-11T10:42:33.439 INFO:teuthology.orchestra.run.smithi050.stdout: mon osd reporter subtree level = osd 2023-10-11T10:42:33.439 INFO:teuthology.orchestra.run.smithi050.stdout: mon osd prime pg temp = true 2023-10-11T10:42:33.439 INFO:teuthology.orchestra.run.smithi050.stdout: mon reweight min bytes per osd = 10 2023-10-11T10:42:33.439 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-11T10:42:33.439 INFO:teuthology.orchestra.run.smithi050.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-10-11T10:42:33.440 INFO:teuthology.orchestra.run.smithi050.stdout: auth mon ticket ttl = 660 # 11m 2023-10-11T10:42:33.440 INFO:teuthology.orchestra.run.smithi050.stdout: auth service ticket ttl = 240 # 4m 2023-10-11T10:42:33.440 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-11T10:42:33.440 INFO:teuthology.orchestra.run.smithi050.stdout: # don't complain about insecure global_id in the test suite 2023-10-11T10:42:33.440 INFO:teuthology.orchestra.run.smithi050.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-10-11T10:42:33.440 INFO:teuthology.orchestra.run.smithi050.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-10-11T10:42:33.441 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-11T10:42:33.441 INFO:teuthology.orchestra.run.smithi050.stdout: # 1m isn't quite enough 2023-10-11T10:42:33.441 INFO:teuthology.orchestra.run.smithi050.stdout: mon_down_mkfs_grace = 2m 2023-10-11T10:42:33.441 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-11T10:42:33.441 INFO:teuthology.orchestra.run.smithi050.stdout: mon_warn_on_filestore_osds = false 2023-10-11T10:42:33.441 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-11T10:42:33.442 INFO:teuthology.orchestra.run.smithi050.stdout:[client] 2023-10-11T10:42:33.442 INFO:teuthology.orchestra.run.smithi050.stdout: rgw cache enabled = true 2023-10-11T10:42:33.442 INFO:teuthology.orchestra.run.smithi050.stdout: rgw enable ops log = true 2023-10-11T10:42:33.442 INFO:teuthology.orchestra.run.smithi050.stdout: rgw enable usage log = true 2023-10-11T10:42:33.442 INFO:teuthology.orchestra.run.smithi050.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-10-11T10:42:33.442 INFO:teuthology.orchestra.run.smithi050.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-10-11T10:42:33.443 INFO:teuthology.orchestra.run.smithi050.stdout: debug rbd = 20 2023-10-11T10:42:33.443 INFO:teuthology.orchestra.run.smithi050.stdout: debug rbd_mirror = 15 2023-10-11T10:42:33.443 INFO:teuthology.orchestra.run.smithi050.stdout: log to stderr = False 2023-10-11T10:42:33.443 INFO:teuthology.orchestra.run.smithi050.stdout: rbd default features = 125 2023-10-11T10:42:33.443 INFO:teuthology.orchestra.run.smithi050.stdout: rbd mirror image policy type = none 2023-10-11T10:42:33.443 INFO:teuthology.orchestra.run.smithi050.stdout:[mon.a] 2023-10-11T10:42:33.444 INFO:teuthology.orchestra.run.smithi050.stdout:[client.mirror.0] 2023-10-11T10:42:33.444 INFO:teuthology.orchestra.run.smithi050.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-10-11T10:42:33.444 INFO:teuthology.orchestra.run.smithi050.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-10-11T10:42:33.444 INFO:teuthology.orchestra.run.smithi050.stdout:[client.mirror.1] 2023-10-11T10:42:33.444 INFO:teuthology.orchestra.run.smithi050.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-10-11T10:42:33.444 INFO:teuthology.orchestra.run.smithi050.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-10-11T10:42:33.444 INFO:teuthology.orchestra.run.smithi050.stdout:[client.mirror.2] 2023-10-11T10:42:33.445 INFO:teuthology.orchestra.run.smithi050.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-10-11T10:42:33.445 INFO:teuthology.orchestra.run.smithi050.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-10-11T10:42:33.445 INFO:teuthology.orchestra.run.smithi050.stdout:[client.mirror.3] 2023-10-11T10:42:33.445 INFO:teuthology.orchestra.run.smithi050.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-10-11T10:42:33.445 INFO:teuthology.orchestra.run.smithi050.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-10-11T10:42:33.447 INFO:teuthology.orchestra.run.smithi111.stdout:[global] 2023-10-11T10:42:33.447 INFO:teuthology.orchestra.run.smithi111.stdout: chdir = "" 2023-10-11T10:42:33.447 INFO:teuthology.orchestra.run.smithi111.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-10-11T10:42:33.447 INFO:teuthology.orchestra.run.smithi111.stdout: auth supported = cephx 2023-10-11T10:42:33.447 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T10:42:33.448 INFO:teuthology.orchestra.run.smithi111.stdout: filestore xattr use omap = true 2023-10-11T10:42:33.448 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T10:42:33.448 INFO:teuthology.orchestra.run.smithi111.stdout: mon clock drift allowed = 1.000 2023-10-11T10:42:33.448 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T10:42:33.448 INFO:teuthology.orchestra.run.smithi111.stdout: osd crush chooseleaf type = 0 2023-10-11T10:42:33.448 INFO:teuthology.orchestra.run.smithi111.stdout: auth debug = true 2023-10-11T10:42:33.449 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T10:42:33.449 INFO:teuthology.orchestra.run.smithi111.stdout: ms die on old message = true 2023-10-11T10:42:33.449 INFO:teuthology.orchestra.run.smithi111.stdout: ms die on bug = true 2023-10-11T10:42:33.449 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T10:42:33.449 INFO:teuthology.orchestra.run.smithi111.stdout: mon max pg per osd = 10000 # >= luminous 2023-10-11T10:42:33.449 INFO:teuthology.orchestra.run.smithi111.stdout: mon pg warn max object skew = 0 2023-10-11T10:42:33.450 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T10:42:33.450 INFO:teuthology.orchestra.run.smithi111.stdout: # disable pg_autoscaler by default for new pools 2023-10-11T10:42:33.450 INFO:teuthology.orchestra.run.smithi111.stdout: osd_pool_default_pg_autoscale_mode = off 2023-10-11T10:42:33.450 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T10:42:33.450 INFO:teuthology.orchestra.run.smithi111.stdout: osd pool default size = 2 2023-10-11T10:42:33.450 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T10:42:33.451 INFO:teuthology.orchestra.run.smithi111.stdout: mon osd allow primary affinity = true 2023-10-11T10:42:33.451 INFO:teuthology.orchestra.run.smithi111.stdout: mon osd allow pg remap = true 2023-10-11T10:42:33.451 INFO:teuthology.orchestra.run.smithi111.stdout: mon warn on legacy crush tunables = false 2023-10-11T10:42:33.451 INFO:teuthology.orchestra.run.smithi111.stdout: mon warn on crush straw calc version zero = false 2023-10-11T10:42:33.451 INFO:teuthology.orchestra.run.smithi111.stdout: mon warn on no sortbitwise = false 2023-10-11T10:42:33.451 INFO:teuthology.orchestra.run.smithi111.stdout: mon warn on osd down out interval zero = false 2023-10-11T10:42:33.452 INFO:teuthology.orchestra.run.smithi111.stdout: mon warn on too few osds = false 2023-10-11T10:42:33.452 INFO:teuthology.orchestra.run.smithi111.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-10-11T10:42:33.452 INFO:teuthology.orchestra.run.smithi111.stdout: mon_warn_on_pool_no_redundancy = false 2023-10-11T10:42:33.452 INFO:teuthology.orchestra.run.smithi111.stdout: mon_allow_pool_size_one = true 2023-10-11T10:42:33.452 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T10:42:33.452 INFO:teuthology.orchestra.run.smithi111.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-10-11T10:42:33.453 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T10:42:33.453 INFO:teuthology.orchestra.run.smithi111.stdout: osd default data pool replay window = 5 2023-10-11T10:42:33.453 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T10:42:33.453 INFO:teuthology.orchestra.run.smithi111.stdout: mon allow pool delete = true 2023-10-11T10:42:33.453 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T10:42:33.453 INFO:teuthology.orchestra.run.smithi111.stdout: mon cluster log file level = debug 2023-10-11T10:42:33.454 INFO:teuthology.orchestra.run.smithi111.stdout: debug asserts on shutdown = true 2023-10-11T10:42:33.454 INFO:teuthology.orchestra.run.smithi111.stdout: mon health detail to clog = false 2023-10-11T10:42:33.454 INFO:teuthology.orchestra.run.smithi111.stdout: mon host = 172.21.15.50 2023-10-11T10:42:33.454 INFO:teuthology.orchestra.run.smithi111.stdout: mon client directed command retry = 5 2023-10-11T10:42:33.454 INFO:teuthology.orchestra.run.smithi111.stdout: mon warn on pool no app = False 2023-10-11T10:42:33.454 INFO:teuthology.orchestra.run.smithi111.stdout: ms inject socket failures = 5000 2023-10-11T10:42:33.455 INFO:teuthology.orchestra.run.smithi111.stdout: fsid = 62c62aca-a3d3-4cfb-94c6-84fa1a529031 2023-10-11T10:42:33.455 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T10:42:33.455 INFO:teuthology.orchestra.run.smithi111.stdout:[osd] 2023-10-11T10:42:33.455 INFO:teuthology.orchestra.run.smithi111.stdout: osd journal size = 100 2023-10-11T10:42:33.455 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T10:42:33.456 INFO:teuthology.orchestra.run.smithi111.stdout: osd scrub load threshold = 5.0 2023-10-11T10:42:33.456 INFO:teuthology.orchestra.run.smithi111.stdout: osd scrub max interval = 600 2023-10-11T10:42:33.456 INFO:teuthology.orchestra.run.smithi111.stdout: osd mclock profile = high_recovery_ops 2023-10-11T10:42:33.456 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T10:42:33.456 INFO:teuthology.orchestra.run.smithi111.stdout: osd recover clone overlap = true 2023-10-11T10:42:33.456 INFO:teuthology.orchestra.run.smithi111.stdout: osd recovery max chunk = 1048576 2023-10-11T10:42:33.457 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T10:42:33.457 INFO:teuthology.orchestra.run.smithi111.stdout: osd debug shutdown = true 2023-10-11T10:42:33.457 INFO:teuthology.orchestra.run.smithi111.stdout: osd debug op order = true 2023-10-11T10:42:33.457 INFO:teuthology.orchestra.run.smithi111.stdout: osd debug verify stray on activate = true 2023-10-11T10:42:33.457 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T10:42:33.458 INFO:teuthology.orchestra.run.smithi111.stdout: osd open classes on start = true 2023-10-11T10:42:33.458 INFO:teuthology.orchestra.run.smithi111.stdout: osd debug pg log writeout = true 2023-10-11T10:42:33.458 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T10:42:33.458 INFO:teuthology.orchestra.run.smithi111.stdout: osd deep scrub update digest min age = 30 2023-10-11T10:42:33.458 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T10:42:33.458 INFO:teuthology.orchestra.run.smithi111.stdout: osd map max advance = 10 2023-10-11T10:42:33.459 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T10:42:33.459 INFO:teuthology.orchestra.run.smithi111.stdout: journal zero on create = true 2023-10-11T10:42:33.459 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T10:42:33.459 INFO:teuthology.orchestra.run.smithi111.stdout: filestore ondisk finisher threads = 3 2023-10-11T10:42:33.459 INFO:teuthology.orchestra.run.smithi111.stdout: filestore apply finisher threads = 3 2023-10-11T10:42:33.459 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T10:42:33.460 INFO:teuthology.orchestra.run.smithi111.stdout: bdev debug aio = true 2023-10-11T10:42:33.460 INFO:teuthology.orchestra.run.smithi111.stdout: osd debug misdirected ops = true 2023-10-11T10:42:33.460 INFO:teuthology.orchestra.run.smithi111.stdout: bdev async discard = True 2023-10-11T10:42:33.460 INFO:teuthology.orchestra.run.smithi111.stdout: bdev enable discard = True 2023-10-11T10:42:33.460 INFO:teuthology.orchestra.run.smithi111.stdout: bluestore allocator = stupid 2023-10-11T10:42:33.461 INFO:teuthology.orchestra.run.smithi111.stdout: bluestore block size = 96636764160 2023-10-11T10:42:33.461 INFO:teuthology.orchestra.run.smithi111.stdout: bluestore fsck on mount = True 2023-10-11T10:42:33.461 INFO:teuthology.orchestra.run.smithi111.stdout: debug bluefs = 1/20 2023-10-11T10:42:33.461 INFO:teuthology.orchestra.run.smithi111.stdout: debug bluestore = 1/20 2023-10-11T10:42:33.461 INFO:teuthology.orchestra.run.smithi111.stdout: debug ms = 1 2023-10-11T10:42:33.461 INFO:teuthology.orchestra.run.smithi111.stdout: debug osd = 20 2023-10-11T10:42:33.462 INFO:teuthology.orchestra.run.smithi111.stdout: debug rocksdb = 4/10 2023-10-11T10:42:33.462 INFO:teuthology.orchestra.run.smithi111.stdout: mon osd backfillfull_ratio = 0.85 2023-10-11T10:42:33.462 INFO:teuthology.orchestra.run.smithi111.stdout: mon osd full ratio = 0.9 2023-10-11T10:42:33.462 INFO:teuthology.orchestra.run.smithi111.stdout: mon osd nearfull ratio = 0.8 2023-10-11T10:42:33.462 INFO:teuthology.orchestra.run.smithi111.stdout: osd failsafe full ratio = 0.95 2023-10-11T10:42:33.462 INFO:teuthology.orchestra.run.smithi111.stdout: osd objectstore = bluestore 2023-10-11T10:42:33.463 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T10:42:33.463 INFO:teuthology.orchestra.run.smithi111.stdout:[mgr] 2023-10-11T10:42:33.463 INFO:teuthology.orchestra.run.smithi111.stdout: debug ms = 1 2023-10-11T10:42:33.463 INFO:teuthology.orchestra.run.smithi111.stdout: debug mgr = 20 2023-10-11T10:42:33.463 INFO:teuthology.orchestra.run.smithi111.stdout: debug mon = 20 2023-10-11T10:42:33.464 INFO:teuthology.orchestra.run.smithi111.stdout: debug auth = 20 2023-10-11T10:42:33.464 INFO:teuthology.orchestra.run.smithi111.stdout: mon reweight min pgs per osd = 4 2023-10-11T10:42:33.464 INFO:teuthology.orchestra.run.smithi111.stdout: mon reweight min bytes per osd = 10 2023-10-11T10:42:33.464 INFO:teuthology.orchestra.run.smithi111.stdout: mgr/telemetry/nag = false 2023-10-11T10:42:33.464 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T10:42:33.464 INFO:teuthology.orchestra.run.smithi111.stdout:[mon] 2023-10-11T10:42:33.465 INFO:teuthology.orchestra.run.smithi111.stdout: debug ms = 1 2023-10-11T10:42:33.465 INFO:teuthology.orchestra.run.smithi111.stdout: debug mon = 20 2023-10-11T10:42:33.465 INFO:teuthology.orchestra.run.smithi111.stdout: debug paxos = 20 2023-10-11T10:42:33.465 INFO:teuthology.orchestra.run.smithi111.stdout: debug auth = 20 2023-10-11T10:42:33.465 INFO:teuthology.orchestra.run.smithi111.stdout: mon data avail warn = 5 2023-10-11T10:42:33.465 INFO:teuthology.orchestra.run.smithi111.stdout: mon mgr mkfs grace = 240 2023-10-11T10:42:33.466 INFO:teuthology.orchestra.run.smithi111.stdout: mon reweight min pgs per osd = 4 2023-10-11T10:42:33.466 INFO:teuthology.orchestra.run.smithi111.stdout: mon osd reporter subtree level = osd 2023-10-11T10:42:33.466 INFO:teuthology.orchestra.run.smithi111.stdout: mon osd prime pg temp = true 2023-10-11T10:42:33.466 INFO:teuthology.orchestra.run.smithi111.stdout: mon reweight min bytes per osd = 10 2023-10-11T10:42:33.467 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T10:42:33.467 INFO:teuthology.orchestra.run.smithi111.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-10-11T10:42:33.467 INFO:teuthology.orchestra.run.smithi111.stdout: auth mon ticket ttl = 660 # 11m 2023-10-11T10:42:33.467 INFO:teuthology.orchestra.run.smithi111.stdout: auth service ticket ttl = 240 # 4m 2023-10-11T10:42:33.467 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T10:42:33.467 INFO:teuthology.orchestra.run.smithi111.stdout: # don't complain about insecure global_id in the test suite 2023-10-11T10:42:33.468 INFO:teuthology.orchestra.run.smithi111.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-10-11T10:42:33.468 INFO:teuthology.orchestra.run.smithi111.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-10-11T10:42:33.468 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T10:42:33.468 INFO:teuthology.orchestra.run.smithi111.stdout: # 1m isn't quite enough 2023-10-11T10:42:33.468 INFO:teuthology.orchestra.run.smithi111.stdout: mon_down_mkfs_grace = 2m 2023-10-11T10:42:33.468 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T10:42:33.469 INFO:teuthology.orchestra.run.smithi111.stdout: mon_warn_on_filestore_osds = false 2023-10-11T10:42:33.469 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T10:42:33.469 INFO:teuthology.orchestra.run.smithi111.stdout:[client] 2023-10-11T10:42:33.469 INFO:teuthology.orchestra.run.smithi111.stdout: rgw cache enabled = true 2023-10-11T10:42:33.469 INFO:teuthology.orchestra.run.smithi111.stdout: rgw enable ops log = true 2023-10-11T10:42:33.470 INFO:teuthology.orchestra.run.smithi111.stdout: rgw enable usage log = true 2023-10-11T10:42:33.470 INFO:teuthology.orchestra.run.smithi111.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-10-11T10:42:33.470 INFO:teuthology.orchestra.run.smithi111.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-10-11T10:42:33.470 INFO:teuthology.orchestra.run.smithi111.stdout: debug rbd = 20 2023-10-11T10:42:33.470 INFO:teuthology.orchestra.run.smithi111.stdout: debug rbd_mirror = 15 2023-10-11T10:42:33.470 INFO:teuthology.orchestra.run.smithi111.stdout: log to stderr = False 2023-10-11T10:42:33.471 INFO:teuthology.orchestra.run.smithi111.stdout: rbd default features = 125 2023-10-11T10:42:33.471 INFO:teuthology.orchestra.run.smithi111.stdout: rbd mirror image policy type = none 2023-10-11T10:42:33.471 INFO:teuthology.orchestra.run.smithi111.stdout:[mon.a] 2023-10-11T10:42:33.471 INFO:teuthology.orchestra.run.smithi111.stdout:[client.mirror.0] 2023-10-11T10:42:33.471 INFO:teuthology.orchestra.run.smithi111.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-10-11T10:42:33.471 INFO:teuthology.orchestra.run.smithi111.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-10-11T10:42:33.472 INFO:teuthology.orchestra.run.smithi111.stdout:[client.mirror.1] 2023-10-11T10:42:33.472 INFO:teuthology.orchestra.run.smithi111.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-10-11T10:42:33.472 INFO:teuthology.orchestra.run.smithi111.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-10-11T10:42:33.472 INFO:teuthology.orchestra.run.smithi111.stdout:[client.mirror.2] 2023-10-11T10:42:33.472 INFO:teuthology.orchestra.run.smithi111.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-10-11T10:42:33.472 INFO:teuthology.orchestra.run.smithi111.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-10-11T10:42:33.473 INFO:teuthology.orchestra.run.smithi111.stdout:[client.mirror.3] 2023-10-11T10:42:33.473 INFO:teuthology.orchestra.run.smithi111.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-10-11T10:42:33.473 INFO:teuthology.orchestra.run.smithi111.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-10-11T10:42:33.473 INFO:tasks.ceph:Creating admin key on cluster1.mon.a... 2023-10-11T10:42:33.473 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/cluster1.keyring 2023-10-11T10:42:33.545 INFO:tasks.ceph:Copying monmap to all nodes... 2023-10-11T10:42:33.545 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2023-10-11T10:42:33.545 DEBUG:teuthology.orchestra.run.smithi050:> dd if=/etc/ceph/cluster1.keyring of=/dev/stdout 2023-10-11T10:42:33.562 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2023-10-11T10:42:33.562 DEBUG:teuthology.orchestra.run.smithi050:> dd if=/home/ubuntu/cephtest/cluster1.monmap of=/dev/stdout 2023-10-11T10:42:33.619 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi050.front.sepia.ceph.com 2023-10-11T10:42:33.620 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2023-10-11T10:42:33.620 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd of=/etc/ceph/cluster1.keyring 2023-10-11T10:42:33.620 DEBUG:teuthology.orchestra.run.smithi050:> sudo chmod 0644 /etc/ceph/cluster1.keyring 2023-10-11T10:42:33.697 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2023-10-11T10:42:33.697 DEBUG:teuthology.orchestra.run.smithi050:> dd of=/home/ubuntu/cephtest/cluster1.monmap 2023-10-11T10:42:33.753 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi111.front.sepia.ceph.com 2023-10-11T10:42:33.754 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2023-10-11T10:42:33.754 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd of=/etc/ceph/cluster1.keyring 2023-10-11T10:42:33.754 DEBUG:teuthology.orchestra.run.smithi111:> sudo chmod 0644 /etc/ceph/cluster1.keyring 2023-10-11T10:42:33.797 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2023-10-11T10:42:33.797 DEBUG:teuthology.orchestra.run.smithi111:> dd of=/home/ubuntu/cephtest/cluster1.monmap 2023-10-11T10:42:33.855 INFO:tasks.ceph:Setting up mon nodes... 2023-10-11T10:42:33.856 INFO:tasks.ceph:Setting up mgr nodes... 2023-10-11T10:42:33.856 DEBUG:teuthology.orchestra.run.smithi050:> sudo mkdir -p /var/lib/ceph/mgr/cluster1-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/cluster1-x/keyring 2023-10-11T10:42:33.913 INFO:teuthology.orchestra.run.smithi050.stdout:creating /var/lib/ceph/mgr/cluster1-x/keyring 2023-10-11T10:42:33.916 INFO:tasks.ceph:Setting up mds nodes... 2023-10-11T10:42:33.916 INFO:tasks.ceph_client:Setting up client nodes... 2023-10-11T10:42:33.917 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/cluster1.client.0.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.0.keyring 2023-10-11T10:42:33.961 INFO:teuthology.orchestra.run.smithi050.stdout:creating /etc/ceph/cluster1.client.0.keyring 2023-10-11T10:42:33.974 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror /etc/ceph/cluster1.client.mirror.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.keyring 2023-10-11T10:42:34.034 INFO:teuthology.orchestra.run.smithi111.stdout:creating /etc/ceph/cluster1.client.mirror.keyring 2023-10-11T10:42:34.048 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.0 /etc/ceph/cluster1.client.mirror.0.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.0.keyring 2023-10-11T10:42:34.094 INFO:teuthology.orchestra.run.smithi111.stdout:creating /etc/ceph/cluster1.client.mirror.0.keyring 2023-10-11T10:42:34.107 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.1 /etc/ceph/cluster1.client.mirror.1.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.1.keyring 2023-10-11T10:42:34.150 INFO:teuthology.orchestra.run.smithi111.stdout:creating /etc/ceph/cluster1.client.mirror.1.keyring 2023-10-11T10:42:34.163 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.2 /etc/ceph/cluster1.client.mirror.2.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.2.keyring 2023-10-11T10:42:34.246 INFO:teuthology.orchestra.run.smithi111.stdout:creating /etc/ceph/cluster1.client.mirror.2.keyring 2023-10-11T10:42:34.259 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.3 /etc/ceph/cluster1.client.mirror.3.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.3.keyring 2023-10-11T10:42:34.341 INFO:teuthology.orchestra.run.smithi111.stdout:creating /etc/ceph/cluster1.client.mirror.3.keyring 2023-10-11T10:42:34.354 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.4 /etc/ceph/cluster1.client.mirror.4.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.4.keyring 2023-10-11T10:42:34.437 INFO:teuthology.orchestra.run.smithi111.stdout:creating /etc/ceph/cluster1.client.mirror.4.keyring 2023-10-11T10:42:34.450 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.5 /etc/ceph/cluster1.client.mirror.5.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.5.keyring 2023-10-11T10:42:34.492 INFO:teuthology.orchestra.run.smithi111.stdout:creating /etc/ceph/cluster1.client.mirror.5.keyring 2023-10-11T10:42:34.505 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.6 /etc/ceph/cluster1.client.mirror.6.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.6.keyring 2023-10-11T10:42:34.588 INFO:teuthology.orchestra.run.smithi111.stdout:creating /etc/ceph/cluster1.client.mirror.6.keyring 2023-10-11T10:42:34.601 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-10-11T10:42:34.601 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi050.front.sepia.ceph.com'): {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'}} 2023-10-11T10:42:34.602 DEBUG:teuthology.orchestra.run.smithi050:> sudo mkdir -p /var/lib/ceph/osd/cluster1-0 2023-10-11T10:42:34.631 INFO:tasks.ceph:roles_to_devs: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2023-10-11T10:42:34.632 INFO:tasks.ceph:role: cluster1.osd.0 2023-10-11T10:42:34.632 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi050.front.sepia.ceph.com 2023-10-11T10:42:34.632 DEBUG:teuthology.orchestra.run.smithi050:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-10-11T10:42:34.701 INFO:teuthology.orchestra.run.smithi050.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-10-11T10:42:34.701 INFO:teuthology.orchestra.run.smithi050.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-11T10:42:34.701 INFO:teuthology.orchestra.run.smithi050.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-11T10:42:34.702 INFO:teuthology.orchestra.run.smithi050.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-11T10:42:34.702 INFO:teuthology.orchestra.run.smithi050.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-11T10:42:34.702 INFO:teuthology.orchestra.run.smithi050.stdout: = sunit=0 swidth=0 blks 2023-10-11T10:42:34.702 INFO:teuthology.orchestra.run.smithi050.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-11T10:42:34.702 INFO:teuthology.orchestra.run.smithi050.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-11T10:42:34.702 INFO:teuthology.orchestra.run.smithi050.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-11T10:42:34.703 INFO:teuthology.orchestra.run.smithi050.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-11T10:42:34.724 INFO:teuthology.orchestra.run.smithi050.stdout:Discarding blocks...Done. 2023-10-11T10:42:34.726 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi050.front.sepia.ceph.com -o noatime 2023-10-11T10:42:34.726 DEBUG:teuthology.orchestra.run.smithi050:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/cluster1-0 2023-10-11T10:42:34.807 DEBUG:teuthology.orchestra.run.smithi050:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster1-0 2023-10-11T10:42:34.880 DEBUG:teuthology.orchestra.run.smithi050:> sudo mkdir -p /var/lib/ceph/osd/cluster1-1 2023-10-11T10:42:34.947 INFO:tasks.ceph:roles_to_devs: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2023-10-11T10:42:34.947 INFO:tasks.ceph:role: cluster1.osd.1 2023-10-11T10:42:34.947 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi050.front.sepia.ceph.com 2023-10-11T10:42:34.948 DEBUG:teuthology.orchestra.run.smithi050:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-10-11T10:42:35.013 INFO:teuthology.orchestra.run.smithi050.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-10-11T10:42:35.013 INFO:teuthology.orchestra.run.smithi050.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-11T10:42:35.014 INFO:teuthology.orchestra.run.smithi050.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-11T10:42:35.014 INFO:teuthology.orchestra.run.smithi050.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-11T10:42:35.014 INFO:teuthology.orchestra.run.smithi050.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-11T10:42:35.014 INFO:teuthology.orchestra.run.smithi050.stdout: = sunit=0 swidth=0 blks 2023-10-11T10:42:35.014 INFO:teuthology.orchestra.run.smithi050.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-11T10:42:35.015 INFO:teuthology.orchestra.run.smithi050.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-11T10:42:35.015 INFO:teuthology.orchestra.run.smithi050.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-11T10:42:35.015 INFO:teuthology.orchestra.run.smithi050.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-11T10:42:35.035 INFO:teuthology.orchestra.run.smithi050.stdout:Discarding blocks...Done. 2023-10-11T10:42:35.037 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi050.front.sepia.ceph.com -o noatime 2023-10-11T10:42:35.037 DEBUG:teuthology.orchestra.run.smithi050:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/cluster1-1 2023-10-11T10:42:35.118 DEBUG:teuthology.orchestra.run.smithi050:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster1-1 2023-10-11T10:42:35.189 DEBUG:teuthology.orchestra.run.smithi050:> sudo mkdir -p /var/lib/ceph/osd/cluster1-2 2023-10-11T10:42:35.256 INFO:tasks.ceph:roles_to_devs: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2023-10-11T10:42:35.256 INFO:tasks.ceph:role: cluster1.osd.2 2023-10-11T10:42:35.256 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi050.front.sepia.ceph.com 2023-10-11T10:42:35.257 DEBUG:teuthology.orchestra.run.smithi050:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-10-11T10:42:35.322 INFO:teuthology.orchestra.run.smithi050.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-10-11T10:42:35.322 INFO:teuthology.orchestra.run.smithi050.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-11T10:42:35.322 INFO:teuthology.orchestra.run.smithi050.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-11T10:42:35.322 INFO:teuthology.orchestra.run.smithi050.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-11T10:42:35.323 INFO:teuthology.orchestra.run.smithi050.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-11T10:42:35.323 INFO:teuthology.orchestra.run.smithi050.stdout: = sunit=0 swidth=0 blks 2023-10-11T10:42:35.323 INFO:teuthology.orchestra.run.smithi050.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-11T10:42:35.323 INFO:teuthology.orchestra.run.smithi050.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-11T10:42:35.323 INFO:teuthology.orchestra.run.smithi050.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-11T10:42:35.324 INFO:teuthology.orchestra.run.smithi050.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-11T10:42:35.344 INFO:teuthology.orchestra.run.smithi050.stdout:Discarding blocks...Done. 2023-10-11T10:42:35.346 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi050.front.sepia.ceph.com -o noatime 2023-10-11T10:42:35.346 DEBUG:teuthology.orchestra.run.smithi050:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/cluster1-2 2023-10-11T10:42:35.425 DEBUG:teuthology.orchestra.run.smithi050:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster1-2 2023-10-11T10:42:35.496 DEBUG:teuthology.orchestra.run.smithi050:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster1 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/cluster1.monmap 2023-10-11T10:42:35.601 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-11T10:42:35.600+0000 7fe582e2f640 -1 auth: error reading file: /var/lib/ceph/osd/cluster1-0/keyring: can't open /var/lib/ceph/osd/cluster1-0/keyring: (2) No such file or directory 2023-10-11T10:42:35.601 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-11T10:42:35.600+0000 7fe582e2f640 -1 created new key in keyring /var/lib/ceph/osd/cluster1-0/keyring 2023-10-11T10:42:35.601 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-11T10:42:35.600+0000 7fe582e2f640 -1 bluestore(/var/lib/ceph/osd/cluster1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-0/block: (2) No such file or directory 2023-10-11T10:42:35.602 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-11T10:42:35.600+0000 7fe582e2f640 -1 bluestore(/var/lib/ceph/osd/cluster1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-0/block: (2) No such file or directory 2023-10-11T10:42:35.602 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-11T10:42:35.600+0000 7fe582e2f640 -1 bluestore(/var/lib/ceph/osd/cluster1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-0/block: (2) No such file or directory 2023-10-11T10:42:35.602 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-11T10:42:35.600+0000 7fe582e2f640 -1 bluestore(/var/lib/ceph/osd/cluster1-0) _read_fsid unparsable uuid 2023-10-11T10:42:38.474 DEBUG:teuthology.orchestra.run.smithi050:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster1-0 2023-10-11T10:42:38.516 DEBUG:teuthology.orchestra.run.smithi050:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster1 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/cluster1.monmap 2023-10-11T10:42:38.611 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-11T10:42:38.610+0000 7fd54342f640 -1 auth: error reading file: /var/lib/ceph/osd/cluster1-1/keyring: can't open /var/lib/ceph/osd/cluster1-1/keyring: (2) No such file or directory 2023-10-11T10:42:38.612 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-11T10:42:38.610+0000 7fd54342f640 -1 created new key in keyring /var/lib/ceph/osd/cluster1-1/keyring 2023-10-11T10:42:38.612 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-11T10:42:38.610+0000 7fd54342f640 -1 bluestore(/var/lib/ceph/osd/cluster1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-1/block: (2) No such file or directory 2023-10-11T10:42:38.612 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-11T10:42:38.610+0000 7fd54342f640 -1 bluestore(/var/lib/ceph/osd/cluster1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-1/block: (2) No such file or directory 2023-10-11T10:42:38.612 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-11T10:42:38.610+0000 7fd54342f640 -1 bluestore(/var/lib/ceph/osd/cluster1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-1/block: (2) No such file or directory 2023-10-11T10:42:38.613 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-11T10:42:38.610+0000 7fd54342f640 -1 bluestore(/var/lib/ceph/osd/cluster1-1) _read_fsid unparsable uuid 2023-10-11T10:42:41.470 DEBUG:teuthology.orchestra.run.smithi050:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster1-1 2023-10-11T10:42:41.503 DEBUG:teuthology.orchestra.run.smithi050:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster1 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/cluster1.monmap 2023-10-11T10:42:41.596 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-11T10:42:41.595+0000 7f7070793640 -1 auth: error reading file: /var/lib/ceph/osd/cluster1-2/keyring: can't open /var/lib/ceph/osd/cluster1-2/keyring: (2) No such file or directory 2023-10-11T10:42:41.596 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-11T10:42:41.595+0000 7f7070793640 -1 created new key in keyring /var/lib/ceph/osd/cluster1-2/keyring 2023-10-11T10:42:41.596 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-11T10:42:41.595+0000 7f7070793640 -1 bluestore(/var/lib/ceph/osd/cluster1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-2/block: (2) No such file or directory 2023-10-11T10:42:41.597 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-11T10:42:41.595+0000 7f7070793640 -1 bluestore(/var/lib/ceph/osd/cluster1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-2/block: (2) No such file or directory 2023-10-11T10:42:41.597 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-11T10:42:41.595+0000 7f7070793640 -1 bluestore(/var/lib/ceph/osd/cluster1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-2/block: (2) No such file or directory 2023-10-11T10:42:41.597 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-11T10:42:41.595+0000 7f7070793640 -1 bluestore(/var/lib/ceph/osd/cluster1-2) _read_fsid unparsable uuid 2023-10-11T10:42:44.456 DEBUG:teuthology.orchestra.run.smithi050:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster1-2 2023-10-11T10:42:44.490 INFO:tasks.ceph:Reading keys from all nodes... 2023-10-11T10:42:44.490 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2023-10-11T10:42:44.490 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd if=/var/lib/ceph/mgr/cluster1-x/keyring of=/dev/stdout 2023-10-11T10:42:44.560 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2023-10-11T10:42:44.560 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd if=/var/lib/ceph/osd/cluster1-0/keyring of=/dev/stdout 2023-10-11T10:42:44.630 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2023-10-11T10:42:44.630 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd if=/var/lib/ceph/osd/cluster1-1/keyring of=/dev/stdout 2023-10-11T10:42:44.700 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2023-10-11T10:42:44.700 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd if=/var/lib/ceph/osd/cluster1-2/keyring of=/dev/stdout 2023-10-11T10:42:44.768 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2023-10-11T10:42:44.768 DEBUG:teuthology.orchestra.run.smithi050:> dd if=/etc/ceph/cluster1.client.0.keyring of=/dev/stdout 2023-10-11T10:42:44.825 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2023-10-11T10:42:44.825 DEBUG:teuthology.orchestra.run.smithi111:> dd if=/etc/ceph/cluster1.client.mirror.keyring of=/dev/stdout 2023-10-11T10:42:44.845 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2023-10-11T10:42:44.846 DEBUG:teuthology.orchestra.run.smithi111:> dd if=/etc/ceph/cluster1.client.mirror.0.keyring of=/dev/stdout 2023-10-11T10:42:44.905 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2023-10-11T10:42:44.905 DEBUG:teuthology.orchestra.run.smithi111:> dd if=/etc/ceph/cluster1.client.mirror.1.keyring of=/dev/stdout 2023-10-11T10:42:44.963 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2023-10-11T10:42:44.963 DEBUG:teuthology.orchestra.run.smithi111:> dd if=/etc/ceph/cluster1.client.mirror.2.keyring of=/dev/stdout 2023-10-11T10:42:45.022 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2023-10-11T10:42:45.022 DEBUG:teuthology.orchestra.run.smithi111:> dd if=/etc/ceph/cluster1.client.mirror.3.keyring of=/dev/stdout 2023-10-11T10:42:45.079 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2023-10-11T10:42:45.080 DEBUG:teuthology.orchestra.run.smithi111:> dd if=/etc/ceph/cluster1.client.mirror.4.keyring of=/dev/stdout 2023-10-11T10:42:45.137 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2023-10-11T10:42:45.138 DEBUG:teuthology.orchestra.run.smithi111:> dd if=/etc/ceph/cluster1.client.mirror.5.keyring of=/dev/stdout 2023-10-11T10:42:45.195 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2023-10-11T10:42:45.195 DEBUG:teuthology.orchestra.run.smithi111:> dd if=/etc/ceph/cluster1.client.mirror.6.keyring of=/dev/stdout 2023-10-11T10:42:45.253 INFO:tasks.ceph:Adding keys to all mons... 2023-10-11T10:42:45.253 DEBUG:teuthology.orchestra.run.smithi050:> sudo tee -a /etc/ceph/cluster1.keyring 2023-10-11T10:42:45.285 INFO:teuthology.orchestra.run.smithi050.stdout:[mgr.x] 2023-10-11T10:42:45.285 INFO:teuthology.orchestra.run.smithi050.stdout: key = AQAZfCZlaXqANhAAOOKmXUoxurY08zTe4P7rrw== 2023-10-11T10:42:45.285 INFO:teuthology.orchestra.run.smithi050.stdout:[osd.0] 2023-10-11T10:42:45.285 INFO:teuthology.orchestra.run.smithi050.stdout: key = AQAbfCZlv+/XIxAAkPJ2XBym67Hhb+6ms1HOWA== 2023-10-11T10:42:45.285 INFO:teuthology.orchestra.run.smithi050.stdout:[osd.1] 2023-10-11T10:42:45.286 INFO:teuthology.orchestra.run.smithi050.stdout: key = AQAefCZlPWVzJBAAtdstICOcp16j9EgtN9+dzA== 2023-10-11T10:42:45.286 INFO:teuthology.orchestra.run.smithi050.stdout:[osd.2] 2023-10-11T10:42:45.286 INFO:teuthology.orchestra.run.smithi050.stdout: key = AQAhfCZlRQaLIxAA4te9LUj34w5VTvLeUiDNzg== 2023-10-11T10:42:45.286 INFO:teuthology.orchestra.run.smithi050.stdout:[client.0] 2023-10-11T10:42:45.287 INFO:teuthology.orchestra.run.smithi050.stdout: key = AQAZfCZlfulXORAAfmZt5Wa8W+HzCL1H5js/YQ== 2023-10-11T10:42:45.287 INFO:teuthology.orchestra.run.smithi050.stdout:[client.mirror] 2023-10-11T10:42:45.287 INFO:teuthology.orchestra.run.smithi050.stdout: key = AQAafCZlDiQWAhAAw8ISseoz3kZ2nXMhyW3FOA== 2023-10-11T10:42:45.287 INFO:teuthology.orchestra.run.smithi050.stdout:[client.mirror.0] 2023-10-11T10:42:45.287 INFO:teuthology.orchestra.run.smithi050.stdout: key = AQAafCZllj+pBRAAVfVJ0RH73M5kzEGU+iZccw== 2023-10-11T10:42:45.287 INFO:teuthology.orchestra.run.smithi050.stdout:[client.mirror.1] 2023-10-11T10:42:45.288 INFO:teuthology.orchestra.run.smithi050.stdout: key = AQAafCZl9/0DCRAA7gTZ6+bifx6zF/lYSzic4Q== 2023-10-11T10:42:45.288 INFO:teuthology.orchestra.run.smithi050.stdout:[client.mirror.2] 2023-10-11T10:42:45.288 INFO:teuthology.orchestra.run.smithi050.stdout: key = AQAafCZlxMOzDhAAkRbydTLt94oJ8u1ymnwzHg== 2023-10-11T10:42:45.288 INFO:teuthology.orchestra.run.smithi050.stdout:[client.mirror.3] 2023-10-11T10:42:45.288 INFO:teuthology.orchestra.run.smithi050.stdout: key = AQAafCZlG+ljFBAASq0jElIuWWFBidZPhhmgow== 2023-10-11T10:42:45.288 INFO:teuthology.orchestra.run.smithi050.stdout:[client.mirror.4] 2023-10-11T10:42:45.289 INFO:teuthology.orchestra.run.smithi050.stdout: key = AQAafCZl2s0aGhAAkKi9RfGF4Mt5HJc6gp6U9Q== 2023-10-11T10:42:45.289 INFO:teuthology.orchestra.run.smithi050.stdout:[client.mirror.5] 2023-10-11T10:42:45.289 INFO:teuthology.orchestra.run.smithi050.stdout: key = AQAafCZljdhmHRAAgpxcvOpRNggOx2ZAwEGT7Q== 2023-10-11T10:42:45.289 INFO:teuthology.orchestra.run.smithi050.stdout:[client.mirror.6] 2023-10-11T10:42:45.289 INFO:teuthology.orchestra.run.smithi050.stdout: key = AQAafCZld2MfIxAALsIP47V/PEERAnGnhIPrlQ== 2023-10-11T10:42:45.290 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-10-11T10:42:45.390 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-11T10:42:45.446 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-11T10:42:45.502 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-11T10:42:45.556 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-11T10:42:45.613 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-11T10:42:45.670 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-11T10:42:45.728 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-11T10:42:45.785 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-11T10:42:45.844 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-11T10:42:45.902 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.4 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-11T10:42:45.959 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.5 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-11T10:42:46.016 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.6 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-11T10:42:46.074 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-10-11T10:42:46.074 DEBUG:teuthology.orchestra.run.smithi050:> sudo mkdir -p /var/lib/ceph/mon/cluster1-a 2023-10-11T10:42:46.101 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster cluster1 --mkfs -i a --monmap /home/ubuntu/cephtest/cluster1.monmap --keyring /etc/ceph/cluster1.keyring 2023-10-11T10:42:46.201 DEBUG:teuthology.orchestra.run.smithi050:> sudo chown -R ceph:ceph /var/lib/ceph/mon/cluster1-a 2023-10-11T10:42:46.227 DEBUG:teuthology.orchestra.run.smithi050:> rm -- /home/ubuntu/cephtest/cluster1.monmap 2023-10-11T10:42:46.284 INFO:tasks.ceph:Starting mon daemons in cluster cluster1... 2023-10-11T10:42:46.284 INFO:tasks.ceph.cluster1.mon.a:Restarting daemon 2023-10-11T10:42:46.285 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster cluster1 -i a 2023-10-11T10:42:46.326 INFO:tasks.ceph.cluster1.mon.a:Started 2023-10-11T10:42:46.326 INFO:tasks.ceph:Starting mgr daemons in cluster cluster1... 2023-10-11T10:42:46.326 INFO:tasks.ceph.cluster1.mgr.x:Restarting daemon 2023-10-11T10:42:46.327 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster cluster1 -i x 2023-10-11T10:42:46.329 INFO:tasks.ceph.cluster1.mgr.x:Started 2023-10-11T10:42:46.329 INFO:tasks.ceph:Setting crush tunables to default 2023-10-11T10:42:46.330 DEBUG:teuthology.orchestra.run.smithi050:> sudo ceph --cluster cluster1 osd crush tunables default 2023-10-11T10:42:46.543 INFO:teuthology.orchestra.run.smithi050.stderr:adjusted tunables profile to default 2023-10-11T10:42:46.554 INFO:tasks.ceph:check_enable_crimson: False 2023-10-11T10:42:46.554 INFO:tasks.ceph:Starting osd daemons in cluster cluster1... 2023-10-11T10:42:46.554 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2023-10-11T10:42:46.555 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd if=/var/lib/ceph/osd/cluster1-0/fsid of=/dev/stdout 2023-10-11T10:42:46.581 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2023-10-11T10:42:46.582 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd if=/var/lib/ceph/osd/cluster1-1/fsid of=/dev/stdout 2023-10-11T10:42:46.649 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2023-10-11T10:42:46.649 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd if=/var/lib/ceph/osd/cluster1-2/fsid of=/dev/stdout 2023-10-11T10:42:46.716 DEBUG:teuthology.orchestra.run.smithi050:> sudo ceph --cluster cluster1 osd new 4a4611ce-7da6-470a-8649-b63d56727239 0 2023-10-11T10:42:46.730 INFO:tasks.ceph.cluster1.mgr.x.smithi050.stderr:2023-10-11T10:42:46.730+0000 7fd06354e200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-11T10:42:46.800 INFO:tasks.ceph.cluster1.mgr.x.smithi050.stderr:2023-10-11T10:42:46.799+0000 7fd06354e200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-11T10:42:46.883 INFO:tasks.ceph.cluster1.mgr.x.smithi050.stderr:2023-10-11T10:42:46.884+0000 7fd06354e200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-11T10:42:46.949 INFO:teuthology.orchestra.run.smithi050.stdout:0 2023-10-11T10:42:46.962 DEBUG:teuthology.orchestra.run.smithi050:> sudo ceph --cluster cluster1 osd new 49a99803-504b-40ed-9ab9-62d95fe81988 1 2023-10-11T10:42:46.967 INFO:tasks.ceph.cluster1.mgr.x.smithi050.stderr:2023-10-11T10:42:46.968+0000 7fd06354e200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-11T10:42:47.121 INFO:tasks.ceph.cluster1.mgr.x.smithi050.stderr:2023-10-11T10:42:47.121+0000 7fd06354e200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-11T10:42:47.180 INFO:teuthology.orchestra.run.smithi050.stdout:1 2023-10-11T10:42:47.193 DEBUG:teuthology.orchestra.run.smithi050:> sudo ceph --cluster cluster1 osd new 69abfaca-41cc-4e5b-91c5-98dfa3531012 2 2023-10-11T10:42:47.450 INFO:teuthology.orchestra.run.smithi050.stdout:2 2023-10-11T10:42:47.463 INFO:tasks.ceph.cluster1.osd.0:Restarting daemon 2023-10-11T10:42:47.463 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster1 -i 0 2023-10-11T10:42:47.465 INFO:tasks.ceph.cluster1.osd.0:Started 2023-10-11T10:42:47.465 INFO:tasks.ceph.cluster1.osd.1:Restarting daemon 2023-10-11T10:42:47.466 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster1 -i 1 2023-10-11T10:42:47.468 INFO:tasks.ceph.cluster1.osd.1:Started 2023-10-11T10:42:47.468 INFO:tasks.ceph.cluster1.osd.2:Restarting daemon 2023-10-11T10:42:47.468 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster1 -i 2 2023-10-11T10:42:47.470 INFO:tasks.ceph.cluster1.osd.2:Started 2023-10-11T10:42:47.471 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2023-10-11T10:42:47.547 INFO:tasks.ceph.cluster1.mgr.x.smithi050.stderr:2023-10-11T10:42:47.547+0000 7fd06354e200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-11T10:42:47.680 INFO:tasks.ceph.cluster1.mgr.x.smithi050.stderr:2023-10-11T10:42:47.679+0000 7fd06354e200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-11T10:42:47.704 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-11T10:42:47.705 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":5,"fsid":"62c62aca-a3d3-4cfb-94c6-84fa1a529031","created":"2023-10-11T10:42:46.445350+0000","modified":"2023-10-11T10:42:47.449931+0000","last_up_change":"0.000000","last_in_change":"2023-10-11T10:42:47.449931+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4a4611ce-7da6-470a-8649-b63d56727239","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"49a99803-504b-40ed-9ab9-62d95fe81988","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"69abfaca-41cc-4e5b-91c5-98dfa3531012","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-11T10:42:47.718 INFO:tasks.ceph.ceph_manager.cluster1:[] 2023-10-11T10:42:47.718 INFO:tasks.ceph:Waiting for OSDs to come up 2023-10-11T10:42:47.773 INFO:tasks.ceph.cluster1.mgr.x.smithi050.stderr:2023-10-11T10:42:47.774+0000 7fd06354e200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-11T10:42:47.858 INFO:tasks.ceph.cluster1.mgr.x.smithi050.stderr:2023-10-11T10:42:47.859+0000 7fd06354e200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-11T10:42:48.020 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster1 osd dump --format=json 2023-10-11T10:42:48.087 INFO:tasks.ceph.cluster1.osd.1.smithi050.stderr:2023-10-11T10:42:48.087+0000 7fededc9b640 -1 Falling back to public interface 2023-10-11T10:42:48.098 INFO:tasks.ceph.cluster1.osd.2.smithi050.stderr:2023-10-11T10:42:48.098+0000 7f3dad493640 -1 Falling back to public interface 2023-10-11T10:42:48.106 INFO:tasks.ceph.cluster1.osd.0.smithi050.stderr:2023-10-11T10:42:48.106+0000 7f61df28b640 -1 Falling back to public interface 2023-10-11T10:42:48.215 INFO:teuthology.misc.health.smithi050.stdout: 2023-10-11T10:42:48.215 INFO:teuthology.misc.health.smithi050.stdout:{"epoch":5,"fsid":"62c62aca-a3d3-4cfb-94c6-84fa1a529031","created":"2023-10-11T10:42:46.445350+0000","modified":"2023-10-11T10:42:47.449931+0000","last_up_change":"0.000000","last_in_change":"2023-10-11T10:42:47.449931+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4a4611ce-7da6-470a-8649-b63d56727239","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"49a99803-504b-40ed-9ab9-62d95fe81988","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"69abfaca-41cc-4e5b-91c5-98dfa3531012","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-11T10:42:48.227 DEBUG:teuthology.misc:0 of 3 OSDs are up 2023-10-11T10:42:48.328 INFO:tasks.ceph.cluster1.mgr.x.smithi050.stderr:2023-10-11T10:42:48.328+0000 7fd06354e200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-11T10:42:48.999 INFO:tasks.ceph.cluster1.mgr.x.smithi050.stderr:2023-10-11T10:42:48.999+0000 7fd06354e200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-11T10:42:49.071 INFO:tasks.ceph.cluster1.mgr.x.smithi050.stderr:2023-10-11T10:42:49.071+0000 7fd06354e200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-11T10:42:49.796 INFO:tasks.ceph.cluster1.mgr.x.smithi050.stderr:2023-10-11T10:42:49.796+0000 7fd06354e200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-11T10:42:49.877 INFO:tasks.ceph.cluster1.mgr.x.smithi050.stderr:2023-10-11T10:42:49.877+0000 7fd06354e200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-11T10:42:50.024 INFO:tasks.ceph.cluster1.mgr.x.smithi050.stderr:2023-10-11T10:42:50.024+0000 7fd06354e200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-11T10:42:50.270 INFO:tasks.ceph.cluster1.mgr.x.smithi050.stderr:2023-10-11T10:42:50.271+0000 7fd06354e200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-11T10:42:50.492 INFO:tasks.ceph.cluster1.mgr.x.smithi050.stderr:2023-10-11T10:42:50.492+0000 7fd06354e200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-11T10:42:50.643 INFO:tasks.ceph.cluster1.mgr.x.smithi050.stderr:2023-10-11T10:42:50.643+0000 7fd06354e200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-11T10:42:50.878 INFO:tasks.ceph.cluster1.mgr.x.smithi050.stderr:2023-10-11T10:42:50.878+0000 7fd06354e200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-11T10:42:51.067 INFO:tasks.ceph.cluster1.mgr.x.smithi050.stderr:2023-10-11T10:42:51.067+0000 7fd06354e200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-11T10:42:51.188 INFO:tasks.ceph.cluster1.mgr.x.smithi050.stderr:2023-10-11T10:42:51.188+0000 7fd06354e200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-11T10:42:51.530 INFO:tasks.ceph.cluster1.mgr.x.smithi050.stderr:2023-10-11T10:42:51.530+0000 7fd06354e200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-11T10:42:51.620 INFO:tasks.ceph.cluster1.osd.1.smithi050.stderr:2023-10-11T10:42:51.621+0000 7fededc9b640 -1 osd.1 0 log_to_monitors true 2023-10-11T10:42:51.637 INFO:tasks.ceph.cluster1.osd.2.smithi050.stderr:2023-10-11T10:42:51.637+0000 7f3dad493640 -1 osd.2 0 log_to_monitors true 2023-10-11T10:42:51.651 INFO:tasks.ceph.cluster1.osd.0.smithi050.stderr:2023-10-11T10:42:51.651+0000 7f61df28b640 -1 osd.0 0 log_to_monitors true 2023-10-11T10:42:51.652 INFO:tasks.ceph.cluster1.mgr.x.smithi050.stderr:2023-10-11T10:42:51.652+0000 7fd06354e200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-11T10:42:51.849 INFO:tasks.ceph.cluster1.mgr.x.smithi050.stderr:2023-10-11T10:42:51.849+0000 7fd06354e200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-11T10:42:52.482 INFO:tasks.ceph.cluster1.mgr.x.smithi050.stderr:2023-10-11T10:42:52.482+0000 7fd06354e200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-11T10:42:53.513 INFO:tasks.ceph.cluster1.mgr.x.smithi050.stderr:2023-10-11T10:42:53.513+0000 7fd01b4ad700 -1 mgr.server handle_report got status from non-daemon mon.a 2023-10-11T10:42:54.529 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster1 osd dump --format=json 2023-10-11T10:42:54.655 INFO:tasks.ceph.cluster1.osd.1.smithi050.stderr:2023-10-11T10:42:54.655+0000 7fede1852700 -1 osd.1 0 waiting for initial osdmap 2023-10-11T10:42:54.670 INFO:tasks.ceph.cluster1.osd.1.smithi050.stderr:2023-10-11T10:42:54.670+0000 7fedd9c2d700 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-11T10:42:54.711 INFO:tasks.ceph.cluster1.osd.2.smithi050.stderr:2023-10-11T10:42:54.711+0000 7f3da104a700 -1 osd.2 0 waiting for initial osdmap 2023-10-11T10:42:54.717 INFO:tasks.ceph.cluster1.osd.0.smithi050.stderr:2023-10-11T10:42:54.717+0000 7f61d2e42700 -1 osd.0 0 waiting for initial osdmap 2023-10-11T10:42:54.727 INFO:tasks.ceph.cluster1.osd.2.smithi050.stderr:2023-10-11T10:42:54.728+0000 7f3d99425700 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-11T10:42:54.736 INFO:tasks.ceph.cluster1.osd.0.smithi050.stderr:2023-10-11T10:42:54.736+0000 7f61cb21d700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-11T10:42:54.921 INFO:teuthology.misc.health.smithi050.stdout: 2023-10-11T10:42:54.922 INFO:teuthology.misc.health.smithi050.stdout:{"epoch":7,"fsid":"62c62aca-a3d3-4cfb-94c6-84fa1a529031","created":"2023-10-11T10:42:46.445350+0000","modified":"2023-10-11T10:42:53.419587+0000","last_up_change":"0.000000","last_in_change":"2023-10-11T10:42:47.449931+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4a4611ce-7da6-470a-8649-b63d56727239","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"49a99803-504b-40ed-9ab9-62d95fe81988","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"69abfaca-41cc-4e5b-91c5-98dfa3531012","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-11T10:42:54.934 DEBUG:teuthology.misc:0 of 3 OSDs are up 2023-10-11T10:43:01.238 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster1 osd dump --format=json 2023-10-11T10:43:01.579 INFO:teuthology.misc.health.smithi050.stdout: 2023-10-11T10:43:01.579 INFO:teuthology.misc.health.smithi050.stdout:{"epoch":11,"fsid":"62c62aca-a3d3-4cfb-94c6-84fa1a529031","created":"2023-10-11T10:42:46.445350+0000","modified":"2023-10-11T10:42:58.659015+0000","last_up_change":"2023-10-11T10:42:55.656744+0000","last_in_change":"2023-10-11T10:42:47.449931+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-11T10:42:56.555720+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4a4611ce-7da6-470a-8649-b63d56727239","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":711452231},{"type":"v1","addr":"172.21.15.50:6817","nonce":711452231}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":711452231},{"type":"v1","addr":"172.21.15.50:6819","nonce":711452231}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6822","nonce":711452231},{"type":"v1","addr":"172.21.15.50:6823","nonce":711452231}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":711452231},{"type":"v1","addr":"172.21.15.50:6821","nonce":711452231}]},"public_addr":"172.21.15.50:6817/711452231","cluster_addr":"172.21.15.50:6819/711452231","heartbeat_back_addr":"172.21.15.50:6823/711452231","heartbeat_front_addr":"172.21.15.50:6821/711452231","state":["exists","up"]},{"osd":1,"uuid":"49a99803-504b-40ed-9ab9-62d95fe81988","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":130890563},{"type":"v1","addr":"172.21.15.50:6801","nonce":130890563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":130890563},{"type":"v1","addr":"172.21.15.50:6803","nonce":130890563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":130890563},{"type":"v1","addr":"172.21.15.50:6807","nonce":130890563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":130890563},{"type":"v1","addr":"172.21.15.50:6805","nonce":130890563}]},"public_addr":"172.21.15.50:6801/130890563","cluster_addr":"172.21.15.50:6803/130890563","heartbeat_back_addr":"172.21.15.50:6807/130890563","heartbeat_front_addr":"172.21.15.50:6805/130890563","state":["exists","up"]},{"osd":2,"uuid":"69abfaca-41cc-4e5b-91c5-98dfa3531012","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":1081260863},{"type":"v1","addr":"172.21.15.50:6809","nonce":1081260863}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":1081260863},{"type":"v1","addr":"172.21.15.50:6811","nonce":1081260863}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":1081260863},{"type":"v1","addr":"172.21.15.50:6815","nonce":1081260863}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":1081260863},{"type":"v1","addr":"172.21.15.50:6813","nonce":1081260863}]},"public_addr":"172.21.15.50:6809/1081260863","cluster_addr":"172.21.15.50:6811/1081260863","heartbeat_back_addr":"172.21.15.50:6815/1081260863","heartbeat_front_addr":"172.21.15.50:6813/1081260863","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T10:42:52.641134+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T10:42:52.631329+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T10:42:52.679138+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-11T10:43:01.592 DEBUG:teuthology.misc:3 of 3 OSDs are up 2023-10-11T10:43:01.593 INFO:tasks.ceph:Creating RBD pool 2023-10-11T10:43:01.593 DEBUG:teuthology.orchestra.run.smithi050:> sudo ceph --cluster cluster1 osd pool create rbd 8 2023-10-11T10:43:02.426 INFO:teuthology.orchestra.run.smithi050.stderr:pool 'rbd' created 2023-10-11T10:43:02.440 DEBUG:teuthology.orchestra.run.smithi050:> sudo ceph --cluster cluster1 osd pool application enable rbd rbd --yes-i-really-mean-it 2023-10-11T10:43:03.431 INFO:teuthology.orchestra.run.smithi050.stderr:enabled application 'rbd' on pool 'rbd' 2023-10-11T10:43:03.447 INFO:tasks.ceph:Starting mds daemons in cluster cluster1... 2023-10-11T10:43:03.448 INFO:tasks.ceph:Waiting until cluster1 daemons up and pgs clean... 2023-10-11T10:43:03.448 INFO:tasks.ceph.ceph_manager.cluster1:waiting for mgr available 2023-10-11T10:43:03.448 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 mgr dump --format=json 2023-10-11T10:43:03.449 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-10-11T10:43:03.856 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-11T10:43:03.868 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":5,"active_gid":4102,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":1509759681},{"type":"v1","addr":"172.21.15.50:6825","nonce":1509759681}]},"active_addr":"172.21.15.50:6825/1509759681","active_change":"2023-10-11T10:42:52.495078+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:0.0.3","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.50:0","nonce":650578107}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.50:0","nonce":2123780004}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.50:0","nonce":2325712257}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.50:0","nonce":1701442157}]}]} 2023-10-11T10:43:03.871 INFO:tasks.ceph.ceph_manager.cluster1:mgr available! 2023-10-11T10:43:03.871 INFO:tasks.ceph.ceph_manager.cluster1:waiting for all up 2023-10-11T10:43:03.872 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2023-10-11T10:43:04.209 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-11T10:43:04.209 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":13,"fsid":"62c62aca-a3d3-4cfb-94c6-84fa1a529031","created":"2023-10-11T10:42:46.445350+0000","modified":"2023-10-11T10:43:03.422504+0000","last_up_change":"2023-10-11T10:42:55.656744+0000","last_in_change":"2023-10-11T10:42:47.449931+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-11T10:42:56.555720+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-11T10:43:01.959424+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4a4611ce-7da6-470a-8649-b63d56727239","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":711452231},{"type":"v1","addr":"172.21.15.50:6817","nonce":711452231}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":711452231},{"type":"v1","addr":"172.21.15.50:6819","nonce":711452231}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6822","nonce":711452231},{"type":"v1","addr":"172.21.15.50:6823","nonce":711452231}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":711452231},{"type":"v1","addr":"172.21.15.50:6821","nonce":711452231}]},"public_addr":"172.21.15.50:6817/711452231","cluster_addr":"172.21.15.50:6819/711452231","heartbeat_back_addr":"172.21.15.50:6823/711452231","heartbeat_front_addr":"172.21.15.50:6821/711452231","state":["exists","up"]},{"osd":1,"uuid":"49a99803-504b-40ed-9ab9-62d95fe81988","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":130890563},{"type":"v1","addr":"172.21.15.50:6801","nonce":130890563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":130890563},{"type":"v1","addr":"172.21.15.50:6803","nonce":130890563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":130890563},{"type":"v1","addr":"172.21.15.50:6807","nonce":130890563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":130890563},{"type":"v1","addr":"172.21.15.50:6805","nonce":130890563}]},"public_addr":"172.21.15.50:6801/130890563","cluster_addr":"172.21.15.50:6803/130890563","heartbeat_back_addr":"172.21.15.50:6807/130890563","heartbeat_front_addr":"172.21.15.50:6805/130890563","state":["exists","up"]},{"osd":2,"uuid":"69abfaca-41cc-4e5b-91c5-98dfa3531012","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":1081260863},{"type":"v1","addr":"172.21.15.50:6809","nonce":1081260863}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":1081260863},{"type":"v1","addr":"172.21.15.50:6811","nonce":1081260863}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":1081260863},{"type":"v1","addr":"172.21.15.50:6815","nonce":1081260863}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":1081260863},{"type":"v1","addr":"172.21.15.50:6813","nonce":1081260863}]},"public_addr":"172.21.15.50:6809/1081260863","cluster_addr":"172.21.15.50:6811/1081260863","heartbeat_back_addr":"172.21.15.50:6815/1081260863","heartbeat_front_addr":"172.21.15.50:6813/1081260863","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T10:42:52.641134+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T10:42:52.631329+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T10:42:52.679138+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-11T10:43:04.223 INFO:tasks.ceph.ceph_manager.cluster1:all up! 2023-10-11T10:43:04.223 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2023-10-11T10:43:04.564 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-11T10:43:04.564 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":14,"fsid":"62c62aca-a3d3-4cfb-94c6-84fa1a529031","created":"2023-10-11T10:42:46.445350+0000","modified":"2023-10-11T10:43:04.424727+0000","last_up_change":"2023-10-11T10:42:55.656744+0000","last_in_change":"2023-10-11T10:42:47.449931+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-11T10:42:56.555720+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-11T10:43:01.959424+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4a4611ce-7da6-470a-8649-b63d56727239","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":711452231},{"type":"v1","addr":"172.21.15.50:6817","nonce":711452231}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":711452231},{"type":"v1","addr":"172.21.15.50:6819","nonce":711452231}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6822","nonce":711452231},{"type":"v1","addr":"172.21.15.50:6823","nonce":711452231}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":711452231},{"type":"v1","addr":"172.21.15.50:6821","nonce":711452231}]},"public_addr":"172.21.15.50:6817/711452231","cluster_addr":"172.21.15.50:6819/711452231","heartbeat_back_addr":"172.21.15.50:6823/711452231","heartbeat_front_addr":"172.21.15.50:6821/711452231","state":["exists","up"]},{"osd":1,"uuid":"49a99803-504b-40ed-9ab9-62d95fe81988","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":130890563},{"type":"v1","addr":"172.21.15.50:6801","nonce":130890563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":130890563},{"type":"v1","addr":"172.21.15.50:6803","nonce":130890563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":130890563},{"type":"v1","addr":"172.21.15.50:6807","nonce":130890563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":130890563},{"type":"v1","addr":"172.21.15.50:6805","nonce":130890563}]},"public_addr":"172.21.15.50:6801/130890563","cluster_addr":"172.21.15.50:6803/130890563","heartbeat_back_addr":"172.21.15.50:6807/130890563","heartbeat_front_addr":"172.21.15.50:6805/130890563","state":["exists","up"]},{"osd":2,"uuid":"69abfaca-41cc-4e5b-91c5-98dfa3531012","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":1081260863},{"type":"v1","addr":"172.21.15.50:6809","nonce":1081260863}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":1081260863},{"type":"v1","addr":"172.21.15.50:6811","nonce":1081260863}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":1081260863},{"type":"v1","addr":"172.21.15.50:6815","nonce":1081260863}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":1081260863},{"type":"v1","addr":"172.21.15.50:6813","nonce":1081260863}]},"public_addr":"172.21.15.50:6809/1081260863","cluster_addr":"172.21.15.50:6811/1081260863","heartbeat_back_addr":"172.21.15.50:6815/1081260863","heartbeat_front_addr":"172.21.15.50:6813/1081260863","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T10:42:52.641134+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T10:42:52.631329+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T10:42:52.679138+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-11T10:43:04.578 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.0 flush_pg_stats 2023-10-11T10:43:04.579 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.1 flush_pg_stats 2023-10-11T10:43:04.579 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.2 flush_pg_stats 2023-10-11T10:43:04.755 INFO:teuthology.orchestra.run.smithi050.stdout:34359738373 2023-10-11T10:43:04.756 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.2 2023-10-11T10:43:04.756 INFO:teuthology.orchestra.run.smithi050.stdout:34359738373 2023-10-11T10:43:04.757 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.0 2023-10-11T10:43:04.758 INFO:teuthology.orchestra.run.smithi050.stdout:34359738371 2023-10-11T10:43:04.759 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.1 2023-10-11T10:43:05.135 INFO:teuthology.orchestra.run.smithi050.stdout:34359738372 2023-10-11T10:43:05.149 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738373 got 34359738372 for osd.2 2023-10-11T10:43:05.152 INFO:teuthology.orchestra.run.smithi050.stdout:34359738372 2023-10-11T10:43:05.167 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738373 got 34359738372 for osd.0 2023-10-11T10:43:05.170 INFO:teuthology.orchestra.run.smithi050.stdout:34359738370 2023-10-11T10:43:05.184 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738371 got 34359738370 for osd.1 2023-10-11T10:43:06.151 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.2 2023-10-11T10:43:06.168 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.0 2023-10-11T10:43:06.186 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.1 2023-10-11T10:43:06.515 INFO:teuthology.orchestra.run.smithi050.stdout:34359738372 2023-10-11T10:43:06.530 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738373 got 34359738372 for osd.2 2023-10-11T10:43:06.551 INFO:teuthology.orchestra.run.smithi050.stdout:34359738370 2023-10-11T10:43:06.559 INFO:teuthology.orchestra.run.smithi050.stdout:34359738372 2023-10-11T10:43:06.570 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738371 got 34359738370 for osd.1 2023-10-11T10:43:06.577 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738373 got 34359738372 for osd.0 2023-10-11T10:43:07.531 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.2 2023-10-11T10:43:07.571 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.1 2023-10-11T10:43:07.577 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.0 2023-10-11T10:43:07.894 INFO:teuthology.orchestra.run.smithi050.stdout:34359738373 2023-10-11T10:43:07.908 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738373 got 34359738373 for osd.2 2023-10-11T10:43:07.909 DEBUG:teuthology.parallel:result is None 2023-10-11T10:43:07.935 INFO:teuthology.orchestra.run.smithi050.stdout:34359738371 2023-10-11T10:43:07.950 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738371 got 34359738371 for osd.1 2023-10-11T10:43:07.950 DEBUG:teuthology.parallel:result is None 2023-10-11T10:43:07.970 INFO:teuthology.orchestra.run.smithi050.stdout:34359738373 2023-10-11T10:43:07.984 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738373 got 34359738373 for osd.0 2023-10-11T10:43:07.984 DEBUG:teuthology.parallel:result is None 2023-10-11T10:43:07.985 INFO:tasks.ceph.ceph_manager.cluster1:waiting for clean 2023-10-11T10:43:07.985 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2023-10-11T10:43:08.358 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-11T10:43:08.359 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2023-10-11T10:43:08.370 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":16,"stamp":"2023-10-11T10:43:06.500770+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":4,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":86732,"kb_used_data":1496,"kb_used_omap":4,"kb_used_meta":85179,"kb_avail":283028788,"statfs":{"total":289910292480,"available":289821478912,"internally_reserved":0,"allocated":1531904,"data_stored":1276633,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4782,"internal_metadata":87223634},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.078116"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-10-11T10:43:03.436044+0000","last_change":"2023-10-11T10:43:03.434569+0000","last_active":"2023-10-11T10:43:03.436044+0000","last_peered":"2023-10-11T10:43:03.436044+0000","last_clean":"2023-10-11T10:43:03.436044+0000","last_became_active":"2023-10-11T10:43:03.434147+0000","last_became_peered":"2023-10-11T10:43:03.434147+0000","last_unstale":"2023-10-11T10:43:03.436044+0000","last_undegraded":"2023-10-11T10:43:03.436044+0000","last_fullsized":"2023-10-11T10:43:03.436044+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:02.419376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:02.419376+0000","last_clean_scrub_stamp":"2023-10-11T10:43:02.419376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T10:57:11.531589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-10-11T10:43:03.435011+0000","last_change":"2023-10-11T10:43:03.432737+0000","last_active":"2023-10-11T10:43:03.435011+0000","last_peered":"2023-10-11T10:43:03.435011+0000","last_clean":"2023-10-11T10:43:03.435011+0000","last_became_active":"2023-10-11T10:43:03.432238+0000","last_became_peered":"2023-10-11T10:43:03.432238+0000","last_unstale":"2023-10-11T10:43:03.435011+0000","last_undegraded":"2023-10-11T10:43:03.435011+0000","last_fullsized":"2023-10-11T10:43:03.435011+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:02.419376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:02.419376+0000","last_clean_scrub_stamp":"2023-10-11T10:43:02.419376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T14:48:50.124706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-10-11T10:43:03.436586+0000","last_change":"2023-10-11T10:43:03.435783+0000","last_active":"2023-10-11T10:43:03.436586+0000","last_peered":"2023-10-11T10:43:03.436586+0000","last_clean":"2023-10-11T10:43:03.436586+0000","last_became_active":"2023-10-11T10:43:03.435469+0000","last_became_peered":"2023-10-11T10:43:03.435469+0000","last_unstale":"2023-10-11T10:43:03.436586+0000","last_undegraded":"2023-10-11T10:43:03.436586+0000","last_fullsized":"2023-10-11T10:43:03.436586+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:02.419376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:02.419376+0000","last_clean_scrub_stamp":"2023-10-11T10:43:02.419376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T11:58:31.329515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-10-11T10:43:03.438365+0000","last_change":"2023-10-11T10:43:03.434801+0000","last_active":"2023-10-11T10:43:03.438365+0000","last_peered":"2023-10-11T10:43:03.438365+0000","last_clean":"2023-10-11T10:43:03.438365+0000","last_became_active":"2023-10-11T10:43:03.434471+0000","last_became_peered":"2023-10-11T10:43:03.434471+0000","last_unstale":"2023-10-11T10:43:03.438365+0000","last_undegraded":"2023-10-11T10:43:03.438365+0000","last_fullsized":"2023-10-11T10:43:03.438365+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:02.419376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:02.419376+0000","last_clean_scrub_stamp":"2023-10-11T10:43:02.419376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T11:29:25.908766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":140,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-11T10:43:02.428398+0000","last_change":"2023-10-11T10:42:57.665559+0000","last_active":"2023-10-11T10:43:02.428398+0000","last_peered":"2023-10-11T10:43:02.428398+0000","last_clean":"2023-10-11T10:43:02.428398+0000","last_became_active":"2023-10-11T10:42:57.664210+0000","last_became_peered":"2023-10-11T10:42:57.664210+0000","last_unstale":"2023-10-11T10:43:02.428398+0000","last_undegraded":"2023-10-11T10:43:02.428398+0000","last_fullsized":"2023-10-11T10:43:02.428398+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:42:56.657464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:42:56.657464+0000","last_clean_scrub_stamp":"2023-10-11T10:42:56.657464+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T15:54:11.939340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-10-11T10:43:03.435039+0000","last_change":"2023-10-11T10:43:03.433261+0000","last_active":"2023-10-11T10:43:03.435039+0000","last_peered":"2023-10-11T10:43:03.435039+0000","last_clean":"2023-10-11T10:43:03.435039+0000","last_became_active":"2023-10-11T10:43:03.432640+0000","last_became_peered":"2023-10-11T10:43:03.432640+0000","last_unstale":"2023-10-11T10:43:03.435039+0000","last_undegraded":"2023-10-11T10:43:03.435039+0000","last_fullsized":"2023-10-11T10:43:03.435039+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:02.419376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:02.419376+0000","last_clean_scrub_stamp":"2023-10-11T10:43:02.419376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:27:19.910157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-10-11T10:43:03.438367+0000","last_change":"2023-10-11T10:43:03.434605+0000","last_active":"2023-10-11T10:43:03.438367+0000","last_peered":"2023-10-11T10:43:03.438367+0000","last_clean":"2023-10-11T10:43:03.438367+0000","last_became_active":"2023-10-11T10:43:03.434104+0000","last_became_peered":"2023-10-11T10:43:03.434104+0000","last_unstale":"2023-10-11T10:43:03.438367+0000","last_undegraded":"2023-10-11T10:43:03.438367+0000","last_fullsized":"2023-10-11T10:43:03.438367+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:02.419376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:02.419376+0000","last_clean_scrub_stamp":"2023-10-11T10:43:02.419376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T16:51:26.878469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-10-11T10:43:04.426852+0000","last_change":"2023-10-11T10:43:03.434716+0000","last_active":"2023-10-11T10:43:04.426852+0000","last_peered":"2023-10-11T10:43:04.426852+0000","last_clean":"2023-10-11T10:43:04.426852+0000","last_became_active":"2023-10-11T10:43:03.434392+0000","last_became_peered":"2023-10-11T10:43:03.434392+0000","last_unstale":"2023-10-11T10:43:04.426852+0000","last_undegraded":"2023-10-11T10:43:04.426852+0000","last_fullsized":"2023-10-11T10:43:04.426852+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:02.419376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:02.419376+0000","last_clean_scrub_stamp":"2023-10-11T10:43:02.419376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T13:50:37.330835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-10-11T10:43:03.435231+0000","last_change":"2023-10-11T10:43:03.433134+0000","last_active":"2023-10-11T10:43:03.435231+0000","last_peered":"2023-10-11T10:43:03.435231+0000","last_clean":"2023-10-11T10:43:03.435231+0000","last_became_active":"2023-10-11T10:43:03.432855+0000","last_became_peered":"2023-10-11T10:43:03.432855+0000","last_unstale":"2023-10-11T10:43:03.435231+0000","last_undegraded":"2023-10-11T10:43:03.435231+0000","last_fullsized":"2023-10-11T10:43:03.435231+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:02.419376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:02.419376+0000","last_clean_scrub_stamp":"2023-10-11T10:43:02.419376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T17:12:17.256198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738373,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27128,"kb_used_data":108,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344712,"statfs":{"total":96636764160,"available":96608985088,"internally_reserved":0,"allocated":110592,"data_stored":30550,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35999999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38600000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738373,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31888,"kb_used_data":696,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339952,"statfs":{"total":96636764160,"available":96604110848,"internally_reserved":0,"allocated":712704,"data_stored":624396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39700000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33300000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738371,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27716,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344124,"statfs":{"total":96636764160,"available":96608382976,"internally_reserved":0,"allocated":708608,"data_stored":621687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.376}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48999999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-11T10:43:08.371 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2023-10-11T10:43:08.704 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-11T10:43:08.707 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2023-10-11T10:43:08.718 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":17,"stamp":"2023-10-11T10:43:08.501137+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":17,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":86748,"kb_used_data":1512,"kb_used_omap":4,"kb_used_meta":85179,"kb_avail":283028772,"statfs":{"total":289910292480,"available":289821462528,"internally_reserved":0,"allocated":1548288,"data_stored":1283589,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4782,"internal_metadata":87223634},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.078483"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-10-11T10:43:05.156018+0000","last_change":"2023-10-11T10:43:03.434569+0000","last_active":"2023-10-11T10:43:05.156018+0000","last_peered":"2023-10-11T10:43:05.156018+0000","last_clean":"2023-10-11T10:43:05.156018+0000","last_became_active":"2023-10-11T10:43:03.434147+0000","last_became_peered":"2023-10-11T10:43:03.434147+0000","last_unstale":"2023-10-11T10:43:05.156018+0000","last_undegraded":"2023-10-11T10:43:05.156018+0000","last_fullsized":"2023-10-11T10:43:05.156018+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:02.419376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:02.419376+0000","last_clean_scrub_stamp":"2023-10-11T10:43:02.419376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:12:12.693291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-10-11T10:43:05.156011+0000","last_change":"2023-10-11T10:43:03.432737+0000","last_active":"2023-10-11T10:43:05.156011+0000","last_peered":"2023-10-11T10:43:05.156011+0000","last_clean":"2023-10-11T10:43:05.156011+0000","last_became_active":"2023-10-11T10:43:03.432238+0000","last_became_peered":"2023-10-11T10:43:03.432238+0000","last_unstale":"2023-10-11T10:43:05.156011+0000","last_undegraded":"2023-10-11T10:43:05.156011+0000","last_fullsized":"2023-10-11T10:43:05.156011+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:02.419376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:02.419376+0000","last_clean_scrub_stamp":"2023-10-11T10:43:02.419376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T20:21:46.997470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-10-11T10:43:05.155993+0000","last_change":"2023-10-11T10:43:03.435783+0000","last_active":"2023-10-11T10:43:05.155993+0000","last_peered":"2023-10-11T10:43:05.155993+0000","last_clean":"2023-10-11T10:43:05.155993+0000","last_became_active":"2023-10-11T10:43:03.435469+0000","last_became_peered":"2023-10-11T10:43:03.435469+0000","last_unstale":"2023-10-11T10:43:05.155993+0000","last_undegraded":"2023-10-11T10:43:05.155993+0000","last_fullsized":"2023-10-11T10:43:05.155993+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:02.419376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:02.419376+0000","last_clean_scrub_stamp":"2023-10-11T10:43:02.419376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T17:08:03.285598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-10-11T10:43:05.155597+0000","last_change":"2023-10-11T10:43:03.434801+0000","last_active":"2023-10-11T10:43:05.155597+0000","last_peered":"2023-10-11T10:43:05.155597+0000","last_clean":"2023-10-11T10:43:05.155597+0000","last_became_active":"2023-10-11T10:43:03.434471+0000","last_became_peered":"2023-10-11T10:43:03.434471+0000","last_unstale":"2023-10-11T10:43:05.155597+0000","last_undegraded":"2023-10-11T10:43:05.155597+0000","last_fullsized":"2023-10-11T10:43:05.155597+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:02.419376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:02.419376+0000","last_clean_scrub_stamp":"2023-10-11T10:43:02.419376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:09:57.094086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":141,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-10-11T10:43:03.431224+0000","last_change":"2023-10-11T10:42:57.665559+0000","last_active":"2023-10-11T10:43:03.431224+0000","last_peered":"2023-10-11T10:43:03.431224+0000","last_clean":"2023-10-11T10:43:03.431224+0000","last_became_active":"2023-10-11T10:42:57.664210+0000","last_became_peered":"2023-10-11T10:42:57.664210+0000","last_unstale":"2023-10-11T10:43:03.431224+0000","last_undegraded":"2023-10-11T10:43:03.431224+0000","last_fullsized":"2023-10-11T10:43:03.431224+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:42:56.657464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:42:56.657464+0000","last_clean_scrub_stamp":"2023-10-11T10:42:56.657464+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T15:54:11.939340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-10-11T10:43:05.155961+0000","last_change":"2023-10-11T10:43:03.433261+0000","last_active":"2023-10-11T10:43:05.155961+0000","last_peered":"2023-10-11T10:43:05.155961+0000","last_clean":"2023-10-11T10:43:05.155961+0000","last_became_active":"2023-10-11T10:43:03.432640+0000","last_became_peered":"2023-10-11T10:43:03.432640+0000","last_unstale":"2023-10-11T10:43:05.155961+0000","last_undegraded":"2023-10-11T10:43:05.155961+0000","last_fullsized":"2023-10-11T10:43:05.155961+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:02.419376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:02.419376+0000","last_clean_scrub_stamp":"2023-10-11T10:43:02.419376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T18:19:21.672854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-10-11T10:43:05.155597+0000","last_change":"2023-10-11T10:43:03.434605+0000","last_active":"2023-10-11T10:43:05.155597+0000","last_peered":"2023-10-11T10:43:05.155597+0000","last_clean":"2023-10-11T10:43:05.155597+0000","last_became_active":"2023-10-11T10:43:03.434104+0000","last_became_peered":"2023-10-11T10:43:03.434104+0000","last_unstale":"2023-10-11T10:43:05.155597+0000","last_undegraded":"2023-10-11T10:43:05.155597+0000","last_fullsized":"2023-10-11T10:43:05.155597+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:02.419376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:02.419376+0000","last_clean_scrub_stamp":"2023-10-11T10:43:02.419376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T13:35:08.276972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-10-11T10:43:04.426852+0000","last_change":"2023-10-11T10:43:03.434716+0000","last_active":"2023-10-11T10:43:04.426852+0000","last_peered":"2023-10-11T10:43:04.426852+0000","last_clean":"2023-10-11T10:43:04.426852+0000","last_became_active":"2023-10-11T10:43:03.434392+0000","last_became_peered":"2023-10-11T10:43:03.434392+0000","last_unstale":"2023-10-11T10:43:04.426852+0000","last_undegraded":"2023-10-11T10:43:04.426852+0000","last_fullsized":"2023-10-11T10:43:04.426852+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:02.419376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:02.419376+0000","last_clean_scrub_stamp":"2023-10-11T10:43:02.419376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T13:50:37.330835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-10-11T10:43:05.155974+0000","last_change":"2023-10-11T10:43:03.433134+0000","last_active":"2023-10-11T10:43:05.155974+0000","last_peered":"2023-10-11T10:43:05.155974+0000","last_clean":"2023-10-11T10:43:05.155974+0000","last_became_active":"2023-10-11T10:43:03.432855+0000","last_became_peered":"2023-10-11T10:43:03.432855+0000","last_unstale":"2023-10-11T10:43:05.155974+0000","last_undegraded":"2023-10-11T10:43:05.155974+0000","last_fullsized":"2023-10-11T10:43:05.155974+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:02.419376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:02.419376+0000","last_clean_scrub_stamp":"2023-10-11T10:43:02.419376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T20:03:03.803743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738374,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27140,"kb_used_data":116,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344700,"statfs":{"total":96636764160,"available":96608972800,"internally_reserved":0,"allocated":118784,"data_stored":34028,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35999999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38600000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738374,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31888,"kb_used_data":696,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339952,"statfs":{"total":96636764160,"available":96604110848,"internally_reserved":0,"allocated":712704,"data_stored":624396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40200000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38200000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738372,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27720,"kb_used_data":700,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344120,"statfs":{"total":96636764160,"available":96608378880,"internally_reserved":0,"allocated":716800,"data_stored":625165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27100000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35799999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-11T10:43:08.719 INFO:tasks.ceph.ceph_manager.cluster1:clean! 2023-10-11T10:43:08.719 INFO:tasks.ceph:Waiting until ceph cluster cluster1 is healthy... 2023-10-11T10:43:08.719 INFO:tasks.ceph.ceph_manager.cluster1:wait_until_healthy 2023-10-11T10:43:08.720 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 health --format=json 2023-10-11T10:43:09.100 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-11T10:43:09.100 INFO:teuthology.orchestra.run.smithi050.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-10-11T10:43:09.114 INFO:tasks.ceph.ceph_manager.cluster1:wait_until_healthy done 2023-10-11T10:43:09.114 INFO:teuthology.run_tasks:Running task ceph... 2023-10-11T10:43:09.126 INFO:tasks.ceph:Creating ceph cluster cluster2... 2023-10-11T10:43:09.127 INFO:tasks.ceph:config {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'none'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'cpu_profile': set(), 'cluster': 'cluster2', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-10-11T10:43:09.127 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/dis-2023-10-11_10:22:06-rbd-wip-dis-testing-distro-default-smithi/7421102', 'branch': 'wip-dis-testing', 'description': 'rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-stupid policy/none rbd-mirror/four-per-cluster supported-random-distro$/{rhel_8} workloads/rbd-mirror-snapshot-stress-workunit-fast-diff}', 'email': 'idryomov@gmail.com', 'first_in_suite': False, 'job_id': '7421102', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'smithi', 'meta': [{'desc': 'run two ceph clusters and install rbd-mirror'}, {'desc': 'configure the permissions for client.mirror'}, {'desc': '2 ceph clusters with 1 mon and 3 osds each'}, {'desc': 'run four rbd-mirror daemons per cluster'}, {'desc': 'run the rbd_mirror_stress.sh workunit to test the rbd-mirror daemon'}], 'name': 'dis-2023-10-11_10:22:06-rbd-wip-dis-testing-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 3, 'size': 30}}], 'os_type': 'rhel', 'os_version': '8.6', 'overrides': {'admin_socket': {'branch': 'wip-dis-testing'}, 'ceph': {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'none'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': '4bf94f59187ed7eba271559c0898cec750ad93d5'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '4bf94f59187ed7eba271559c0898cec750ad93d5'}}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-dis-testing-qa', 'sha1': '3d099a3859cc211cf8dd2acdaf3194ba0e96ea9c'}}, 'owner': 'scheduled_dis@teuthology', 'priority': 74, 'repo': 'https://github.com/ceph/ceph-ci', 'roles': [['cluster1.mon.a', 'cluster1.mgr.x', 'cluster2.mgr.x', 'cluster1.osd.0', 'cluster1.osd.1', 'cluster1.osd.2', 'cluster1.client.0', 'cluster2.client.0'], ['cluster2.mon.a', 'cluster2.osd.0', 'cluster2.osd.1', 'cluster2.osd.2', 'cluster1.client.mirror', 'cluster1.client.mirror.0', 'cluster1.client.mirror.1', 'cluster1.client.mirror.2', 'cluster1.client.mirror.3', 'cluster1.client.mirror.4', 'cluster1.client.mirror.5', 'cluster1.client.mirror.6', 'cluster2.client.mirror', 'cluster2.client.mirror.0', 'cluster2.client.mirror.1', 'cluster2.client.mirror.2', 'cluster2.client.mirror.3', 'cluster2.client.mirror.4', 'cluster2.client.mirror.5', 'cluster2.client.mirror.6']], 'seed': 2449, 'sha1': '4bf94f59187ed7eba271559c0898cec750ad93d5', 'sleep_before_teardown': 0, 'subset': '49/99', 'suite': 'rbd', 'suite_branch': 'wip-dis-testing-qa', 'suite_path': '/home/teuthworker/src/github.com_idryomov_ceph_3d099a3859cc211cf8dd2acdaf3194ba0e96ea9c/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/idryomov/ceph', 'suite_sha1': '3d099a3859cc211cf8dd2acdaf3194ba0e96ea9c', 'targets': {'smithi050.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs=', 'smithi111.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': ['rbd-mirror'], 'flavor': 'default', 'sha1': '4bf94f59187ed7eba271559c0898cec750ad93d5'}}, {'ceph': {'cluster': 'cluster1', 'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'none'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': '4bf94f59187ed7eba271559c0898cec750ad93d5'}}, {'ceph': {'cluster': 'cluster2', 'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'none'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': '4bf94f59187ed7eba271559c0898cec750ad93d5'}}, {'exec': {'cluster1.client.mirror.0': ["sudo ceph --cluster cluster1 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'"]}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.0', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.1', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.2', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.3', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.0', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.1', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.2', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.3', 'thrash': True}}, {'rbd-mirror-thrash': {'cluster': 'cluster1'}}, {'rbd-mirror-thrash': {'cluster': 'cluster2'}}, {'workunit': {'clients': {'cluster1.client.mirror': ['rbd/rbd_mirror_stress.sh']}, 'env': {'CEPH_ARGS': '', 'MIRROR_IMAGE_MODE': 'snapshot', 'MIRROR_POOL_MODE': 'image', 'RBD_IMAGE_FEATURES': 'layering,exclusive-lock,object-map,fast-diff', 'RBD_MIRROR_INSTANCES': '4', 'RBD_MIRROR_USE_EXISTING_CLUSTER': '1', 'RBD_MIRROR_USE_RBD_MIRROR': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '54e62bcbac4e53d9685e08328b790d3b20d71cae', 'timestamp': '2023-10-11_10:22:06', 'tube': 'smithi', 'user': 'dis', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2830'} 2023-10-11T10:43:09.128 DEBUG:teuthology.orchestra.run.smithi050:> install -d -m0755 -- /home/ubuntu/cephtest/cluster2.data 2023-10-11T10:43:09.131 DEBUG:teuthology.orchestra.run.smithi111:> install -d -m0755 -- /home/ubuntu/cephtest/cluster2.data 2023-10-11T10:43:09.151 DEBUG:teuthology.orchestra.run.smithi050:> sudo install -d -m0777 -- /var/run/ceph 2023-10-11T10:43:09.188 DEBUG:teuthology.orchestra.run.smithi111:> sudo install -d -m0777 -- /var/run/ceph 2023-10-11T10:43:09.224 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2023-10-11T10:43:09.224 DEBUG:teuthology.orchestra.run.smithi111:> dd if=/scratch_devs of=/dev/stdout 2023-10-11T10:43:09.284 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-10-11T10:43:09.284 DEBUG:teuthology.orchestra.run.smithi111:> stat /dev/vg_nvme/lv_1 2023-10-11T10:43:09.343 INFO:teuthology.orchestra.run.smithi111.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-10-11T10:43:09.343 INFO:teuthology.orchestra.run.smithi111.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-11T10:43:09.344 INFO:teuthology.orchestra.run.smithi111.stdout:Device: 6h/6d Inode: 391913 Links: 1 2023-10-11T10:43:09.344 INFO:teuthology.orchestra.run.smithi111.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-11T10:43:09.344 INFO:teuthology.orchestra.run.smithi111.stdout:Context: system_u:object_r:device_t:s0 2023-10-11T10:43:09.344 INFO:teuthology.orchestra.run.smithi111.stdout:Access: 2023-10-11 10:41:52.464055521 +0000 2023-10-11T10:43:09.344 INFO:teuthology.orchestra.run.smithi111.stdout:Modify: 2023-10-11 10:36:37.072230862 +0000 2023-10-11T10:43:09.344 INFO:teuthology.orchestra.run.smithi111.stdout:Change: 2023-10-11 10:36:37.072230862 +0000 2023-10-11T10:43:09.345 INFO:teuthology.orchestra.run.smithi111.stdout: Birth: - 2023-10-11T10:43:09.345 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-10-11T10:43:09.411 INFO:teuthology.orchestra.run.smithi111.stderr:1+0 records in 2023-10-11T10:43:09.411 INFO:teuthology.orchestra.run.smithi111.stderr:1+0 records out 2023-10-11T10:43:09.412 INFO:teuthology.orchestra.run.smithi111.stderr:512 bytes copied, 0.000253875 s, 2.0 MB/s 2023-10-11T10:43:09.412 DEBUG:teuthology.orchestra.run.smithi111:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-10-11T10:43:09.471 DEBUG:teuthology.orchestra.run.smithi111:> stat /dev/vg_nvme/lv_2 2023-10-11T10:43:09.530 INFO:teuthology.orchestra.run.smithi111.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-10-11T10:43:09.530 INFO:teuthology.orchestra.run.smithi111.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-11T10:43:09.530 INFO:teuthology.orchestra.run.smithi111.stdout:Device: 6h/6d Inode: 387661 Links: 1 2023-10-11T10:43:09.530 INFO:teuthology.orchestra.run.smithi111.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-11T10:43:09.530 INFO:teuthology.orchestra.run.smithi111.stdout:Context: system_u:object_r:device_t:s0 2023-10-11T10:43:09.530 INFO:teuthology.orchestra.run.smithi111.stdout:Access: 2023-10-11 10:41:52.464055521 +0000 2023-10-11T10:43:09.531 INFO:teuthology.orchestra.run.smithi111.stdout:Modify: 2023-10-11 10:36:37.586221830 +0000 2023-10-11T10:43:09.531 INFO:teuthology.orchestra.run.smithi111.stdout:Change: 2023-10-11 10:36:37.586221830 +0000 2023-10-11T10:43:09.531 INFO:teuthology.orchestra.run.smithi111.stdout: Birth: - 2023-10-11T10:43:09.531 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-10-11T10:43:09.597 INFO:teuthology.orchestra.run.smithi111.stderr:1+0 records in 2023-10-11T10:43:09.597 INFO:teuthology.orchestra.run.smithi111.stderr:1+0 records out 2023-10-11T10:43:09.597 INFO:teuthology.orchestra.run.smithi111.stderr:512 bytes copied, 0.000208387 s, 2.5 MB/s 2023-10-11T10:43:09.598 DEBUG:teuthology.orchestra.run.smithi111:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-10-11T10:43:09.657 DEBUG:teuthology.orchestra.run.smithi111:> stat /dev/vg_nvme/lv_3 2023-10-11T10:43:09.715 INFO:teuthology.orchestra.run.smithi111.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-10-11T10:43:09.715 INFO:teuthology.orchestra.run.smithi111.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-11T10:43:09.716 INFO:teuthology.orchestra.run.smithi111.stdout:Device: 6h/6d Inode: 394730 Links: 1 2023-10-11T10:43:09.716 INFO:teuthology.orchestra.run.smithi111.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-11T10:43:09.716 INFO:teuthology.orchestra.run.smithi111.stdout:Context: system_u:object_r:device_t:s0 2023-10-11T10:43:09.716 INFO:teuthology.orchestra.run.smithi111.stdout:Access: 2023-10-11 10:41:52.464055521 +0000 2023-10-11T10:43:09.716 INFO:teuthology.orchestra.run.smithi111.stdout:Modify: 2023-10-11 10:36:38.104212726 +0000 2023-10-11T10:43:09.716 INFO:teuthology.orchestra.run.smithi111.stdout:Change: 2023-10-11 10:36:38.104212726 +0000 2023-10-11T10:43:09.717 INFO:teuthology.orchestra.run.smithi111.stdout: Birth: - 2023-10-11T10:43:09.717 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-10-11T10:43:09.781 INFO:teuthology.orchestra.run.smithi111.stderr:1+0 records in 2023-10-11T10:43:09.782 INFO:teuthology.orchestra.run.smithi111.stderr:1+0 records out 2023-10-11T10:43:09.782 INFO:teuthology.orchestra.run.smithi111.stderr:512 bytes copied, 0.000211949 s, 2.4 MB/s 2023-10-11T10:43:09.783 DEBUG:teuthology.orchestra.run.smithi111:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-10-11T10:43:09.841 DEBUG:teuthology.orchestra.run.smithi111:> stat /dev/vg_nvme/lv_4 2023-10-11T10:43:09.898 INFO:teuthology.orchestra.run.smithi111.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-10-11T10:43:09.899 INFO:teuthology.orchestra.run.smithi111.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-11T10:43:09.899 INFO:teuthology.orchestra.run.smithi111.stdout:Device: 6h/6d Inode: 389674 Links: 1 2023-10-11T10:43:09.899 INFO:teuthology.orchestra.run.smithi111.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-11T10:43:09.899 INFO:teuthology.orchestra.run.smithi111.stdout:Context: system_u:object_r:device_t:s0 2023-10-11T10:43:09.900 INFO:teuthology.orchestra.run.smithi111.stdout:Access: 2023-10-11 10:41:52.464055521 +0000 2023-10-11T10:43:09.900 INFO:teuthology.orchestra.run.smithi111.stdout:Modify: 2023-10-11 10:36:38.616203729 +0000 2023-10-11T10:43:09.900 INFO:teuthology.orchestra.run.smithi111.stdout:Change: 2023-10-11 10:36:38.616203729 +0000 2023-10-11T10:43:09.900 INFO:teuthology.orchestra.run.smithi111.stdout: Birth: - 2023-10-11T10:43:09.900 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-10-11T10:43:09.964 INFO:teuthology.orchestra.run.smithi111.stderr:1+0 records in 2023-10-11T10:43:09.964 INFO:teuthology.orchestra.run.smithi111.stderr:1+0 records out 2023-10-11T10:43:09.964 INFO:teuthology.orchestra.run.smithi111.stderr:512 bytes copied, 0.000219873 s, 2.3 MB/s 2023-10-11T10:43:09.965 DEBUG:teuthology.orchestra.run.smithi111:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-10-11T10:43:10.023 INFO:tasks.ceph:osd dev map: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2023-10-11T10:43:10.024 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi111.front.sepia.ceph.com'): {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'}} 2023-10-11T10:43:10.024 INFO:tasks.ceph:Generating config... 2023-10-11T10:43:10.025 INFO:tasks.ceph:[client] debug rbd = 20 2023-10-11T10:43:10.026 INFO:tasks.ceph:[client] debug rbd_mirror = 15 2023-10-11T10:43:10.026 INFO:tasks.ceph:[client] log to stderr = False 2023-10-11T10:43:10.026 INFO:tasks.ceph:[client] rbd default features = 125 2023-10-11T10:43:10.027 INFO:tasks.ceph:[client] rbd mirror image policy type = none 2023-10-11T10:43:10.027 INFO:tasks.ceph:[client.mirror.0] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-10-11T10:43:10.027 INFO:tasks.ceph:[client.mirror.0] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-10-11T10:43:10.028 INFO:tasks.ceph:[client.mirror.1] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-10-11T10:43:10.028 INFO:tasks.ceph:[client.mirror.1] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-10-11T10:43:10.028 INFO:tasks.ceph:[client.mirror.2] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-10-11T10:43:10.029 INFO:tasks.ceph:[client.mirror.2] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-10-11T10:43:10.029 INFO:tasks.ceph:[client.mirror.3] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-10-11T10:43:10.029 INFO:tasks.ceph:[client.mirror.3] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-10-11T10:43:10.030 INFO:tasks.ceph:[global] mon client directed command retry = 5 2023-10-11T10:43:10.030 INFO:tasks.ceph:[global] mon warn on pool no app = False 2023-10-11T10:43:10.030 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2023-10-11T10:43:10.030 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-10-11T10:43:10.031 INFO:tasks.ceph:[mgr] debug ms = 1 2023-10-11T10:43:10.031 INFO:tasks.ceph:[mon] debug mon = 20 2023-10-11T10:43:10.031 INFO:tasks.ceph:[mon] debug ms = 1 2023-10-11T10:43:10.032 INFO:tasks.ceph:[mon] debug paxos = 20 2023-10-11T10:43:10.032 INFO:tasks.ceph:[osd] bdev async discard = True 2023-10-11T10:43:10.032 INFO:tasks.ceph:[osd] bdev enable discard = True 2023-10-11T10:43:10.033 INFO:tasks.ceph:[osd] bluestore allocator = stupid 2023-10-11T10:43:10.033 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2023-10-11T10:43:10.033 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2023-10-11T10:43:10.034 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2023-10-11T10:43:10.034 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2023-10-11T10:43:10.034 INFO:tasks.ceph:[osd] debug ms = 1 2023-10-11T10:43:10.035 INFO:tasks.ceph:[osd] debug osd = 20 2023-10-11T10:43:10.035 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2023-10-11T10:43:10.035 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2023-10-11T10:43:10.035 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2023-10-11T10:43:10.036 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2023-10-11T10:43:10.036 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2023-10-11T10:43:10.036 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2023-10-11T10:43:10.037 INFO:tasks.ceph:Setting up cluster2.mon.a... 2023-10-11T10:43:10.037 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/cluster2.keyring 2023-10-11T10:43:10.105 INFO:teuthology.orchestra.run.smithi111.stdout:creating /etc/ceph/cluster2.keyring 2023-10-11T10:43:10.108 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/cluster2.keyring 2023-10-11T10:43:10.194 DEBUG:teuthology.orchestra.run.smithi111:> sudo chmod 0644 /etc/ceph/cluster2.keyring 2023-10-11T10:43:10.220 DEBUG:tasks.ceph:Ceph mon addresses: [('cluster2.mon.a', '172.21.15.111')] 2023-10-11T10:43:10.221 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.111', 'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'none'}, 'mon.a': {}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}} 2023-10-11T10:43:10.221 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2023-10-11T10:43:10.222 DEBUG:teuthology.orchestra.run.smithi111:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-10-11T10:43:10.276 DEBUG:teuthology.orchestra.run.smithi111:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.111 --print /home/ubuntu/cephtest/cluster2.monmap 2023-10-11T10:43:10.357 INFO:teuthology.orchestra.run.smithi111.stdout:monmaptool: monmap file /home/ubuntu/cephtest/cluster2.monmap 2023-10-11T10:43:10.357 INFO:teuthology.orchestra.run.smithi111.stdout:monmaptool: generated fsid 022ce6f2-9a5b-45ad-8b77-a1332d92f228 2023-10-11T10:43:10.357 INFO:teuthology.orchestra.run.smithi111.stdout:setting min_mon_release = pacific 2023-10-11T10:43:10.357 INFO:teuthology.orchestra.run.smithi111.stdout:epoch 0 2023-10-11T10:43:10.358 INFO:teuthology.orchestra.run.smithi111.stdout:fsid 022ce6f2-9a5b-45ad-8b77-a1332d92f228 2023-10-11T10:43:10.358 INFO:teuthology.orchestra.run.smithi111.stdout:last_changed 2023-10-11T10:43:10.357717+0000 2023-10-11T10:43:10.358 INFO:teuthology.orchestra.run.smithi111.stdout:created 2023-10-11T10:43:10.357717+0000 2023-10-11T10:43:10.358 INFO:teuthology.orchestra.run.smithi111.stdout:min_mon_release 16 (pacific) 2023-10-11T10:43:10.358 INFO:teuthology.orchestra.run.smithi111.stdout:election_strategy: 1 2023-10-11T10:43:10.359 INFO:teuthology.orchestra.run.smithi111.stdout:0: [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] mon.a 2023-10-11T10:43:10.359 INFO:teuthology.orchestra.run.smithi111.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/cluster2.monmap (1 monitors) 2023-10-11T10:43:10.360 DEBUG:teuthology.orchestra.run.smithi111:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-10-11T10:43:10.415 INFO:tasks.ceph:Writing /etc/ceph/cluster2.conf for FSID 022ce6f2-9a5b-45ad-8b77-a1332d92f228... 2023-10-11T10:43:10.416 DEBUG:teuthology.orchestra.run.smithi050:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster2.conf && sudo chmod 0644 /etc/ceph/cluster2.conf > /dev/null 2023-10-11T10:43:10.419 DEBUG:teuthology.orchestra.run.smithi111:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster2.conf && sudo chmod 0644 /etc/ceph/cluster2.conf > /dev/null 2023-10-11T10:43:10.473 INFO:teuthology.orchestra.run.smithi050.stdout:[global] 2023-10-11T10:43:10.473 INFO:teuthology.orchestra.run.smithi050.stdout: chdir = "" 2023-10-11T10:43:10.473 INFO:teuthology.orchestra.run.smithi050.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-10-11T10:43:10.473 INFO:teuthology.orchestra.run.smithi050.stdout: auth supported = cephx 2023-10-11T10:43:10.474 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-11T10:43:10.474 INFO:teuthology.orchestra.run.smithi050.stdout: filestore xattr use omap = true 2023-10-11T10:43:10.474 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-11T10:43:10.474 INFO:teuthology.orchestra.run.smithi050.stdout: mon clock drift allowed = 1.000 2023-10-11T10:43:10.474 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-11T10:43:10.475 INFO:teuthology.orchestra.run.smithi050.stdout: osd crush chooseleaf type = 0 2023-10-11T10:43:10.475 INFO:teuthology.orchestra.run.smithi050.stdout: auth debug = true 2023-10-11T10:43:10.475 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-11T10:43:10.475 INFO:teuthology.orchestra.run.smithi050.stdout: ms die on old message = true 2023-10-11T10:43:10.475 INFO:teuthology.orchestra.run.smithi050.stdout: ms die on bug = true 2023-10-11T10:43:10.475 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-11T10:43:10.476 INFO:teuthology.orchestra.run.smithi050.stdout: mon max pg per osd = 10000 # >= luminous 2023-10-11T10:43:10.476 INFO:teuthology.orchestra.run.smithi050.stdout: mon pg warn max object skew = 0 2023-10-11T10:43:10.476 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-11T10:43:10.476 INFO:teuthology.orchestra.run.smithi050.stdout: # disable pg_autoscaler by default for new pools 2023-10-11T10:43:10.476 INFO:teuthology.orchestra.run.smithi050.stdout: osd_pool_default_pg_autoscale_mode = off 2023-10-11T10:43:10.476 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-11T10:43:10.477 INFO:teuthology.orchestra.run.smithi050.stdout: osd pool default size = 2 2023-10-11T10:43:10.477 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-11T10:43:10.477 INFO:teuthology.orchestra.run.smithi050.stdout: mon osd allow primary affinity = true 2023-10-11T10:43:10.477 INFO:teuthology.orchestra.run.smithi050.stdout: mon osd allow pg remap = true 2023-10-11T10:43:10.477 INFO:teuthology.orchestra.run.smithi050.stdout: mon warn on legacy crush tunables = false 2023-10-11T10:43:10.477 INFO:teuthology.orchestra.run.smithi050.stdout: mon warn on crush straw calc version zero = false 2023-10-11T10:43:10.478 INFO:teuthology.orchestra.run.smithi050.stdout: mon warn on no sortbitwise = false 2023-10-11T10:43:10.478 INFO:teuthology.orchestra.run.smithi050.stdout: mon warn on osd down out interval zero = false 2023-10-11T10:43:10.478 INFO:teuthology.orchestra.run.smithi050.stdout: mon warn on too few osds = false 2023-10-11T10:43:10.478 INFO:teuthology.orchestra.run.smithi050.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-10-11T10:43:10.478 INFO:teuthology.orchestra.run.smithi050.stdout: mon_warn_on_pool_no_redundancy = false 2023-10-11T10:43:10.479 INFO:teuthology.orchestra.run.smithi050.stdout: mon_allow_pool_size_one = true 2023-10-11T10:43:10.479 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-11T10:43:10.479 INFO:teuthology.orchestra.run.smithi050.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-10-11T10:43:10.479 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-11T10:43:10.479 INFO:teuthology.orchestra.run.smithi050.stdout: osd default data pool replay window = 5 2023-10-11T10:43:10.479 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-11T10:43:10.480 INFO:teuthology.orchestra.run.smithi050.stdout: mon allow pool delete = true 2023-10-11T10:43:10.480 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-11T10:43:10.480 INFO:teuthology.orchestra.run.smithi050.stdout: mon cluster log file level = debug 2023-10-11T10:43:10.480 INFO:teuthology.orchestra.run.smithi050.stdout: debug asserts on shutdown = true 2023-10-11T10:43:10.480 INFO:teuthology.orchestra.run.smithi050.stdout: mon health detail to clog = false 2023-10-11T10:43:10.480 INFO:teuthology.orchestra.run.smithi050.stdout: mon host = 172.21.15.111 2023-10-11T10:43:10.481 INFO:teuthology.orchestra.run.smithi050.stdout: mon client directed command retry = 5 2023-10-11T10:43:10.481 INFO:teuthology.orchestra.run.smithi050.stdout: mon warn on pool no app = False 2023-10-11T10:43:10.481 INFO:teuthology.orchestra.run.smithi050.stdout: ms inject socket failures = 5000 2023-10-11T10:43:10.481 INFO:teuthology.orchestra.run.smithi050.stdout: fsid = 022ce6f2-9a5b-45ad-8b77-a1332d92f228 2023-10-11T10:43:10.481 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-11T10:43:10.481 INFO:teuthology.orchestra.run.smithi050.stdout:[osd] 2023-10-11T10:43:10.482 INFO:teuthology.orchestra.run.smithi050.stdout: osd journal size = 100 2023-10-11T10:43:10.482 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-11T10:43:10.482 INFO:teuthology.orchestra.run.smithi050.stdout: osd scrub load threshold = 5.0 2023-10-11T10:43:10.482 INFO:teuthology.orchestra.run.smithi050.stdout: osd scrub max interval = 600 2023-10-11T10:43:10.482 INFO:teuthology.orchestra.run.smithi050.stdout: osd mclock profile = high_recovery_ops 2023-10-11T10:43:10.482 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-11T10:43:10.483 INFO:teuthology.orchestra.run.smithi050.stdout: osd recover clone overlap = true 2023-10-11T10:43:10.483 INFO:teuthology.orchestra.run.smithi050.stdout: osd recovery max chunk = 1048576 2023-10-11T10:43:10.483 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-11T10:43:10.483 INFO:teuthology.orchestra.run.smithi050.stdout: osd debug shutdown = true 2023-10-11T10:43:10.483 INFO:teuthology.orchestra.run.smithi050.stdout: osd debug op order = true 2023-10-11T10:43:10.483 INFO:teuthology.orchestra.run.smithi050.stdout: osd debug verify stray on activate = true 2023-10-11T10:43:10.484 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-11T10:43:10.484 INFO:teuthology.orchestra.run.smithi050.stdout: osd open classes on start = true 2023-10-11T10:43:10.484 INFO:teuthology.orchestra.run.smithi050.stdout: osd debug pg log writeout = true 2023-10-11T10:43:10.484 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-11T10:43:10.484 INFO:teuthology.orchestra.run.smithi050.stdout: osd deep scrub update digest min age = 30 2023-10-11T10:43:10.484 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-11T10:43:10.485 INFO:teuthology.orchestra.run.smithi050.stdout: osd map max advance = 10 2023-10-11T10:43:10.485 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-11T10:43:10.485 INFO:teuthology.orchestra.run.smithi050.stdout: journal zero on create = true 2023-10-11T10:43:10.485 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-11T10:43:10.485 INFO:teuthology.orchestra.run.smithi050.stdout: filestore ondisk finisher threads = 3 2023-10-11T10:43:10.485 INFO:teuthology.orchestra.run.smithi050.stdout: filestore apply finisher threads = 3 2023-10-11T10:43:10.486 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-11T10:43:10.486 INFO:teuthology.orchestra.run.smithi050.stdout: bdev debug aio = true 2023-10-11T10:43:10.486 INFO:teuthology.orchestra.run.smithi050.stdout: osd debug misdirected ops = true 2023-10-11T10:43:10.486 INFO:teuthology.orchestra.run.smithi050.stdout: bdev async discard = True 2023-10-11T10:43:10.486 INFO:teuthology.orchestra.run.smithi050.stdout: bdev enable discard = True 2023-10-11T10:43:10.486 INFO:teuthology.orchestra.run.smithi050.stdout: bluestore allocator = stupid 2023-10-11T10:43:10.487 INFO:teuthology.orchestra.run.smithi050.stdout: bluestore block size = 96636764160 2023-10-11T10:43:10.487 INFO:teuthology.orchestra.run.smithi050.stdout: bluestore fsck on mount = True 2023-10-11T10:43:10.487 INFO:teuthology.orchestra.run.smithi050.stdout: debug bluefs = 1/20 2023-10-11T10:43:10.487 INFO:teuthology.orchestra.run.smithi050.stdout: debug bluestore = 1/20 2023-10-11T10:43:10.487 INFO:teuthology.orchestra.run.smithi050.stdout: debug ms = 1 2023-10-11T10:43:10.487 INFO:teuthology.orchestra.run.smithi050.stdout: debug osd = 20 2023-10-11T10:43:10.488 INFO:teuthology.orchestra.run.smithi050.stdout: debug rocksdb = 4/10 2023-10-11T10:43:10.488 INFO:teuthology.orchestra.run.smithi050.stdout: mon osd backfillfull_ratio = 0.85 2023-10-11T10:43:10.488 INFO:teuthology.orchestra.run.smithi050.stdout: mon osd full ratio = 0.9 2023-10-11T10:43:10.488 INFO:teuthology.orchestra.run.smithi050.stdout: mon osd nearfull ratio = 0.8 2023-10-11T10:43:10.488 INFO:teuthology.orchestra.run.smithi050.stdout: osd failsafe full ratio = 0.95 2023-10-11T10:43:10.489 INFO:teuthology.orchestra.run.smithi050.stdout: osd objectstore = bluestore 2023-10-11T10:43:10.489 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-11T10:43:10.489 INFO:teuthology.orchestra.run.smithi050.stdout:[mgr] 2023-10-11T10:43:10.489 INFO:teuthology.orchestra.run.smithi050.stdout: debug ms = 1 2023-10-11T10:43:10.489 INFO:teuthology.orchestra.run.smithi050.stdout: debug mgr = 20 2023-10-11T10:43:10.489 INFO:teuthology.orchestra.run.smithi050.stdout: debug mon = 20 2023-10-11T10:43:10.490 INFO:teuthology.orchestra.run.smithi050.stdout: debug auth = 20 2023-10-11T10:43:10.490 INFO:teuthology.orchestra.run.smithi050.stdout: mon reweight min pgs per osd = 4 2023-10-11T10:43:10.490 INFO:teuthology.orchestra.run.smithi050.stdout: mon reweight min bytes per osd = 10 2023-10-11T10:43:10.490 INFO:teuthology.orchestra.run.smithi050.stdout: mgr/telemetry/nag = false 2023-10-11T10:43:10.490 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-11T10:43:10.491 INFO:teuthology.orchestra.run.smithi050.stdout:[mon] 2023-10-11T10:43:10.491 INFO:teuthology.orchestra.run.smithi050.stdout: debug ms = 1 2023-10-11T10:43:10.491 INFO:teuthology.orchestra.run.smithi050.stdout: debug mon = 20 2023-10-11T10:43:10.491 INFO:teuthology.orchestra.run.smithi050.stdout: debug paxos = 20 2023-10-11T10:43:10.491 INFO:teuthology.orchestra.run.smithi050.stdout: debug auth = 20 2023-10-11T10:43:10.492 INFO:teuthology.orchestra.run.smithi050.stdout: mon data avail warn = 5 2023-10-11T10:43:10.492 INFO:teuthology.orchestra.run.smithi050.stdout: mon mgr mkfs grace = 240 2023-10-11T10:43:10.492 INFO:teuthology.orchestra.run.smithi050.stdout: mon reweight min pgs per osd = 4 2023-10-11T10:43:10.492 INFO:teuthology.orchestra.run.smithi050.stdout: mon osd reporter subtree level = osd 2023-10-11T10:43:10.492 INFO:teuthology.orchestra.run.smithi050.stdout: mon osd prime pg temp = true 2023-10-11T10:43:10.492 INFO:teuthology.orchestra.run.smithi050.stdout: mon reweight min bytes per osd = 10 2023-10-11T10:43:10.493 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-11T10:43:10.493 INFO:teuthology.orchestra.run.smithi050.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-10-11T10:43:10.493 INFO:teuthology.orchestra.run.smithi050.stdout: auth mon ticket ttl = 660 # 11m 2023-10-11T10:43:10.493 INFO:teuthology.orchestra.run.smithi050.stdout: auth service ticket ttl = 240 # 4m 2023-10-11T10:43:10.493 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-11T10:43:10.493 INFO:teuthology.orchestra.run.smithi050.stdout: # don't complain about insecure global_id in the test suite 2023-10-11T10:43:10.494 INFO:teuthology.orchestra.run.smithi050.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-10-11T10:43:10.494 INFO:teuthology.orchestra.run.smithi050.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-10-11T10:43:10.494 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-11T10:43:10.494 INFO:teuthology.orchestra.run.smithi050.stdout: # 1m isn't quite enough 2023-10-11T10:43:10.494 INFO:teuthology.orchestra.run.smithi050.stdout: mon_down_mkfs_grace = 2m 2023-10-11T10:43:10.494 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-11T10:43:10.495 INFO:teuthology.orchestra.run.smithi050.stdout: mon_warn_on_filestore_osds = false 2023-10-11T10:43:10.495 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-11T10:43:10.495 INFO:teuthology.orchestra.run.smithi050.stdout:[client] 2023-10-11T10:43:10.495 INFO:teuthology.orchestra.run.smithi050.stdout: rgw cache enabled = true 2023-10-11T10:43:10.495 INFO:teuthology.orchestra.run.smithi050.stdout: rgw enable ops log = true 2023-10-11T10:43:10.495 INFO:teuthology.orchestra.run.smithi050.stdout: rgw enable usage log = true 2023-10-11T10:43:10.496 INFO:teuthology.orchestra.run.smithi050.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-10-11T10:43:10.496 INFO:teuthology.orchestra.run.smithi050.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-10-11T10:43:10.496 INFO:teuthology.orchestra.run.smithi050.stdout: debug rbd = 20 2023-10-11T10:43:10.496 INFO:teuthology.orchestra.run.smithi050.stdout: debug rbd_mirror = 15 2023-10-11T10:43:10.496 INFO:teuthology.orchestra.run.smithi050.stdout: log to stderr = False 2023-10-11T10:43:10.497 INFO:teuthology.orchestra.run.smithi050.stdout: rbd default features = 125 2023-10-11T10:43:10.497 INFO:teuthology.orchestra.run.smithi050.stdout: rbd mirror image policy type = none 2023-10-11T10:43:10.497 INFO:teuthology.orchestra.run.smithi050.stdout:[mon.a] 2023-10-11T10:43:10.497 INFO:teuthology.orchestra.run.smithi050.stdout:[client.mirror.0] 2023-10-11T10:43:10.497 INFO:teuthology.orchestra.run.smithi050.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-10-11T10:43:10.497 INFO:teuthology.orchestra.run.smithi050.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-10-11T10:43:10.498 INFO:teuthology.orchestra.run.smithi050.stdout:[client.mirror.1] 2023-10-11T10:43:10.498 INFO:teuthology.orchestra.run.smithi050.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-10-11T10:43:10.498 INFO:teuthology.orchestra.run.smithi050.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-10-11T10:43:10.498 INFO:teuthology.orchestra.run.smithi050.stdout:[client.mirror.2] 2023-10-11T10:43:10.498 INFO:teuthology.orchestra.run.smithi050.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-10-11T10:43:10.498 INFO:teuthology.orchestra.run.smithi050.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-10-11T10:43:10.499 INFO:teuthology.orchestra.run.smithi050.stdout:[client.mirror.3] 2023-10-11T10:43:10.499 INFO:teuthology.orchestra.run.smithi050.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-10-11T10:43:10.499 INFO:teuthology.orchestra.run.smithi050.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-10-11T10:43:10.500 INFO:teuthology.orchestra.run.smithi111.stdout:[global] 2023-10-11T10:43:10.500 INFO:teuthology.orchestra.run.smithi111.stdout: chdir = "" 2023-10-11T10:43:10.500 INFO:teuthology.orchestra.run.smithi111.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-10-11T10:43:10.500 INFO:teuthology.orchestra.run.smithi111.stdout: auth supported = cephx 2023-10-11T10:43:10.500 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T10:43:10.501 INFO:teuthology.orchestra.run.smithi111.stdout: filestore xattr use omap = true 2023-10-11T10:43:10.501 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T10:43:10.501 INFO:teuthology.orchestra.run.smithi111.stdout: mon clock drift allowed = 1.000 2023-10-11T10:43:10.501 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T10:43:10.501 INFO:teuthology.orchestra.run.smithi111.stdout: osd crush chooseleaf type = 0 2023-10-11T10:43:10.501 INFO:teuthology.orchestra.run.smithi111.stdout: auth debug = true 2023-10-11T10:43:10.502 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T10:43:10.502 INFO:teuthology.orchestra.run.smithi111.stdout: ms die on old message = true 2023-10-11T10:43:10.502 INFO:teuthology.orchestra.run.smithi111.stdout: ms die on bug = true 2023-10-11T10:43:10.502 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T10:43:10.502 INFO:teuthology.orchestra.run.smithi111.stdout: mon max pg per osd = 10000 # >= luminous 2023-10-11T10:43:10.502 INFO:teuthology.orchestra.run.smithi111.stdout: mon pg warn max object skew = 0 2023-10-11T10:43:10.503 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T10:43:10.503 INFO:teuthology.orchestra.run.smithi111.stdout: # disable pg_autoscaler by default for new pools 2023-10-11T10:43:10.503 INFO:teuthology.orchestra.run.smithi111.stdout: osd_pool_default_pg_autoscale_mode = off 2023-10-11T10:43:10.503 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T10:43:10.503 INFO:teuthology.orchestra.run.smithi111.stdout: osd pool default size = 2 2023-10-11T10:43:10.503 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T10:43:10.504 INFO:teuthology.orchestra.run.smithi111.stdout: mon osd allow primary affinity = true 2023-10-11T10:43:10.504 INFO:teuthology.orchestra.run.smithi111.stdout: mon osd allow pg remap = true 2023-10-11T10:43:10.504 INFO:teuthology.orchestra.run.smithi111.stdout: mon warn on legacy crush tunables = false 2023-10-11T10:43:10.504 INFO:teuthology.orchestra.run.smithi111.stdout: mon warn on crush straw calc version zero = false 2023-10-11T10:43:10.504 INFO:teuthology.orchestra.run.smithi111.stdout: mon warn on no sortbitwise = false 2023-10-11T10:43:10.504 INFO:teuthology.orchestra.run.smithi111.stdout: mon warn on osd down out interval zero = false 2023-10-11T10:43:10.505 INFO:teuthology.orchestra.run.smithi111.stdout: mon warn on too few osds = false 2023-10-11T10:43:10.505 INFO:teuthology.orchestra.run.smithi111.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-10-11T10:43:10.505 INFO:teuthology.orchestra.run.smithi111.stdout: mon_warn_on_pool_no_redundancy = false 2023-10-11T10:43:10.505 INFO:teuthology.orchestra.run.smithi111.stdout: mon_allow_pool_size_one = true 2023-10-11T10:43:10.505 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T10:43:10.505 INFO:teuthology.orchestra.run.smithi111.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-10-11T10:43:10.506 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T10:43:10.506 INFO:teuthology.orchestra.run.smithi111.stdout: osd default data pool replay window = 5 2023-10-11T10:43:10.506 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T10:43:10.506 INFO:teuthology.orchestra.run.smithi111.stdout: mon allow pool delete = true 2023-10-11T10:43:10.506 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T10:43:10.507 INFO:teuthology.orchestra.run.smithi111.stdout: mon cluster log file level = debug 2023-10-11T10:43:10.507 INFO:teuthology.orchestra.run.smithi111.stdout: debug asserts on shutdown = true 2023-10-11T10:43:10.507 INFO:teuthology.orchestra.run.smithi111.stdout: mon health detail to clog = false 2023-10-11T10:43:10.507 INFO:teuthology.orchestra.run.smithi111.stdout: mon host = 172.21.15.111 2023-10-11T10:43:10.507 INFO:teuthology.orchestra.run.smithi111.stdout: mon client directed command retry = 5 2023-10-11T10:43:10.507 INFO:teuthology.orchestra.run.smithi111.stdout: mon warn on pool no app = False 2023-10-11T10:43:10.507 INFO:teuthology.orchestra.run.smithi111.stdout: ms inject socket failures = 5000 2023-10-11T10:43:10.508 INFO:teuthology.orchestra.run.smithi111.stdout: fsid = 022ce6f2-9a5b-45ad-8b77-a1332d92f228 2023-10-11T10:43:10.508 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T10:43:10.508 INFO:teuthology.orchestra.run.smithi111.stdout:[osd] 2023-10-11T10:43:10.508 INFO:teuthology.orchestra.run.smithi111.stdout: osd journal size = 100 2023-10-11T10:43:10.508 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T10:43:10.508 INFO:teuthology.orchestra.run.smithi111.stdout: osd scrub load threshold = 5.0 2023-10-11T10:43:10.509 INFO:teuthology.orchestra.run.smithi111.stdout: osd scrub max interval = 600 2023-10-11T10:43:10.509 INFO:teuthology.orchestra.run.smithi111.stdout: osd mclock profile = high_recovery_ops 2023-10-11T10:43:10.509 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T10:43:10.509 INFO:teuthology.orchestra.run.smithi111.stdout: osd recover clone overlap = true 2023-10-11T10:43:10.509 INFO:teuthology.orchestra.run.smithi111.stdout: osd recovery max chunk = 1048576 2023-10-11T10:43:10.509 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T10:43:10.510 INFO:teuthology.orchestra.run.smithi111.stdout: osd debug shutdown = true 2023-10-11T10:43:10.510 INFO:teuthology.orchestra.run.smithi111.stdout: osd debug op order = true 2023-10-11T10:43:10.510 INFO:teuthology.orchestra.run.smithi111.stdout: osd debug verify stray on activate = true 2023-10-11T10:43:10.510 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T10:43:10.510 INFO:teuthology.orchestra.run.smithi111.stdout: osd open classes on start = true 2023-10-11T10:43:10.511 INFO:teuthology.orchestra.run.smithi111.stdout: osd debug pg log writeout = true 2023-10-11T10:43:10.511 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T10:43:10.511 INFO:teuthology.orchestra.run.smithi111.stdout: osd deep scrub update digest min age = 30 2023-10-11T10:43:10.511 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T10:43:10.511 INFO:teuthology.orchestra.run.smithi111.stdout: osd map max advance = 10 2023-10-11T10:43:10.511 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T10:43:10.512 INFO:teuthology.orchestra.run.smithi111.stdout: journal zero on create = true 2023-10-11T10:43:10.512 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T10:43:10.512 INFO:teuthology.orchestra.run.smithi111.stdout: filestore ondisk finisher threads = 3 2023-10-11T10:43:10.512 INFO:teuthology.orchestra.run.smithi111.stdout: filestore apply finisher threads = 3 2023-10-11T10:43:10.512 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T10:43:10.513 INFO:teuthology.orchestra.run.smithi111.stdout: bdev debug aio = true 2023-10-11T10:43:10.513 INFO:teuthology.orchestra.run.smithi111.stdout: osd debug misdirected ops = true 2023-10-11T10:43:10.513 INFO:teuthology.orchestra.run.smithi111.stdout: bdev async discard = True 2023-10-11T10:43:10.513 INFO:teuthology.orchestra.run.smithi111.stdout: bdev enable discard = True 2023-10-11T10:43:10.513 INFO:teuthology.orchestra.run.smithi111.stdout: bluestore allocator = stupid 2023-10-11T10:43:10.513 INFO:teuthology.orchestra.run.smithi111.stdout: bluestore block size = 96636764160 2023-10-11T10:43:10.514 INFO:teuthology.orchestra.run.smithi111.stdout: bluestore fsck on mount = True 2023-10-11T10:43:10.514 INFO:teuthology.orchestra.run.smithi111.stdout: debug bluefs = 1/20 2023-10-11T10:43:10.514 INFO:teuthology.orchestra.run.smithi111.stdout: debug bluestore = 1/20 2023-10-11T10:43:10.514 INFO:teuthology.orchestra.run.smithi111.stdout: debug ms = 1 2023-10-11T10:43:10.514 INFO:teuthology.orchestra.run.smithi111.stdout: debug osd = 20 2023-10-11T10:43:10.514 INFO:teuthology.orchestra.run.smithi111.stdout: debug rocksdb = 4/10 2023-10-11T10:43:10.515 INFO:teuthology.orchestra.run.smithi111.stdout: mon osd backfillfull_ratio = 0.85 2023-10-11T10:43:10.515 INFO:teuthology.orchestra.run.smithi111.stdout: mon osd full ratio = 0.9 2023-10-11T10:43:10.515 INFO:teuthology.orchestra.run.smithi111.stdout: mon osd nearfull ratio = 0.8 2023-10-11T10:43:10.515 INFO:teuthology.orchestra.run.smithi111.stdout: osd failsafe full ratio = 0.95 2023-10-11T10:43:10.515 INFO:teuthology.orchestra.run.smithi111.stdout: osd objectstore = bluestore 2023-10-11T10:43:10.515 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T10:43:10.516 INFO:teuthology.orchestra.run.smithi111.stdout:[mgr] 2023-10-11T10:43:10.516 INFO:teuthology.orchestra.run.smithi111.stdout: debug ms = 1 2023-10-11T10:43:10.516 INFO:teuthology.orchestra.run.smithi111.stdout: debug mgr = 20 2023-10-11T10:43:10.516 INFO:teuthology.orchestra.run.smithi111.stdout: debug mon = 20 2023-10-11T10:43:10.516 INFO:teuthology.orchestra.run.smithi111.stdout: debug auth = 20 2023-10-11T10:43:10.516 INFO:teuthology.orchestra.run.smithi111.stdout: mon reweight min pgs per osd = 4 2023-10-11T10:43:10.517 INFO:teuthology.orchestra.run.smithi111.stdout: mon reweight min bytes per osd = 10 2023-10-11T10:43:10.517 INFO:teuthology.orchestra.run.smithi111.stdout: mgr/telemetry/nag = false 2023-10-11T10:43:10.517 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T10:43:10.517 INFO:teuthology.orchestra.run.smithi111.stdout:[mon] 2023-10-11T10:43:10.517 INFO:teuthology.orchestra.run.smithi111.stdout: debug ms = 1 2023-10-11T10:43:10.517 INFO:teuthology.orchestra.run.smithi111.stdout: debug mon = 20 2023-10-11T10:43:10.518 INFO:teuthology.orchestra.run.smithi111.stdout: debug paxos = 20 2023-10-11T10:43:10.518 INFO:teuthology.orchestra.run.smithi111.stdout: debug auth = 20 2023-10-11T10:43:10.518 INFO:teuthology.orchestra.run.smithi111.stdout: mon data avail warn = 5 2023-10-11T10:43:10.518 INFO:teuthology.orchestra.run.smithi111.stdout: mon mgr mkfs grace = 240 2023-10-11T10:43:10.518 INFO:teuthology.orchestra.run.smithi111.stdout: mon reweight min pgs per osd = 4 2023-10-11T10:43:10.519 INFO:teuthology.orchestra.run.smithi111.stdout: mon osd reporter subtree level = osd 2023-10-11T10:43:10.519 INFO:teuthology.orchestra.run.smithi111.stdout: mon osd prime pg temp = true 2023-10-11T10:43:10.519 INFO:teuthology.orchestra.run.smithi111.stdout: mon reweight min bytes per osd = 10 2023-10-11T10:43:10.519 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T10:43:10.519 INFO:teuthology.orchestra.run.smithi111.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-10-11T10:43:10.519 INFO:teuthology.orchestra.run.smithi111.stdout: auth mon ticket ttl = 660 # 11m 2023-10-11T10:43:10.520 INFO:teuthology.orchestra.run.smithi111.stdout: auth service ticket ttl = 240 # 4m 2023-10-11T10:43:10.520 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T10:43:10.520 INFO:teuthology.orchestra.run.smithi111.stdout: # don't complain about insecure global_id in the test suite 2023-10-11T10:43:10.520 INFO:teuthology.orchestra.run.smithi111.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-10-11T10:43:10.520 INFO:teuthology.orchestra.run.smithi111.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-10-11T10:43:10.520 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T10:43:10.520 INFO:teuthology.orchestra.run.smithi111.stdout: # 1m isn't quite enough 2023-10-11T10:43:10.521 INFO:teuthology.orchestra.run.smithi111.stdout: mon_down_mkfs_grace = 2m 2023-10-11T10:43:10.521 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T10:43:10.521 INFO:teuthology.orchestra.run.smithi111.stdout: mon_warn_on_filestore_osds = false 2023-10-11T10:43:10.521 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T10:43:10.521 INFO:teuthology.orchestra.run.smithi111.stdout:[client] 2023-10-11T10:43:10.522 INFO:teuthology.orchestra.run.smithi111.stdout: rgw cache enabled = true 2023-10-11T10:43:10.522 INFO:teuthology.orchestra.run.smithi111.stdout: rgw enable ops log = true 2023-10-11T10:43:10.522 INFO:teuthology.orchestra.run.smithi111.stdout: rgw enable usage log = true 2023-10-11T10:43:10.522 INFO:teuthology.orchestra.run.smithi111.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-10-11T10:43:10.522 INFO:teuthology.orchestra.run.smithi111.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-10-11T10:43:10.523 INFO:teuthology.orchestra.run.smithi111.stdout: debug rbd = 20 2023-10-11T10:43:10.523 INFO:teuthology.orchestra.run.smithi111.stdout: debug rbd_mirror = 15 2023-10-11T10:43:10.523 INFO:teuthology.orchestra.run.smithi111.stdout: log to stderr = False 2023-10-11T10:43:10.523 INFO:teuthology.orchestra.run.smithi111.stdout: rbd default features = 125 2023-10-11T10:43:10.523 INFO:teuthology.orchestra.run.smithi111.stdout: rbd mirror image policy type = none 2023-10-11T10:43:10.523 INFO:teuthology.orchestra.run.smithi111.stdout:[mon.a] 2023-10-11T10:43:10.524 INFO:teuthology.orchestra.run.smithi111.stdout:[client.mirror.0] 2023-10-11T10:43:10.524 INFO:teuthology.orchestra.run.smithi111.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-10-11T10:43:10.524 INFO:teuthology.orchestra.run.smithi111.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-10-11T10:43:10.524 INFO:teuthology.orchestra.run.smithi111.stdout:[client.mirror.1] 2023-10-11T10:43:10.524 INFO:teuthology.orchestra.run.smithi111.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-10-11T10:43:10.524 INFO:teuthology.orchestra.run.smithi111.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-10-11T10:43:10.525 INFO:teuthology.orchestra.run.smithi111.stdout:[client.mirror.2] 2023-10-11T10:43:10.525 INFO:teuthology.orchestra.run.smithi111.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-10-11T10:43:10.525 INFO:teuthology.orchestra.run.smithi111.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-10-11T10:43:10.525 INFO:teuthology.orchestra.run.smithi111.stdout:[client.mirror.3] 2023-10-11T10:43:10.525 INFO:teuthology.orchestra.run.smithi111.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-10-11T10:43:10.526 INFO:teuthology.orchestra.run.smithi111.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-10-11T10:43:10.527 INFO:tasks.ceph:Creating admin key on cluster2.mon.a... 2023-10-11T10:43:10.527 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/cluster2.keyring 2023-10-11T10:43:10.583 INFO:tasks.ceph:Copying monmap to all nodes... 2023-10-11T10:43:10.583 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2023-10-11T10:43:10.583 DEBUG:teuthology.orchestra.run.smithi111:> dd if=/etc/ceph/cluster2.keyring of=/dev/stdout 2023-10-11T10:43:10.600 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2023-10-11T10:43:10.600 DEBUG:teuthology.orchestra.run.smithi111:> dd if=/home/ubuntu/cephtest/cluster2.monmap of=/dev/stdout 2023-10-11T10:43:10.656 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi050.front.sepia.ceph.com 2023-10-11T10:43:10.657 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2023-10-11T10:43:10.657 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd of=/etc/ceph/cluster2.keyring 2023-10-11T10:43:10.657 DEBUG:teuthology.orchestra.run.smithi050:> sudo chmod 0644 /etc/ceph/cluster2.keyring 2023-10-11T10:43:10.703 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2023-10-11T10:43:10.703 DEBUG:teuthology.orchestra.run.smithi050:> dd of=/home/ubuntu/cephtest/cluster2.monmap 2023-10-11T10:43:10.762 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi111.front.sepia.ceph.com 2023-10-11T10:43:10.762 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2023-10-11T10:43:10.763 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd of=/etc/ceph/cluster2.keyring 2023-10-11T10:43:10.763 DEBUG:teuthology.orchestra.run.smithi111:> sudo chmod 0644 /etc/ceph/cluster2.keyring 2023-10-11T10:43:10.799 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2023-10-11T10:43:10.799 DEBUG:teuthology.orchestra.run.smithi111:> dd of=/home/ubuntu/cephtest/cluster2.monmap 2023-10-11T10:43:10.855 INFO:tasks.ceph:Setting up mon nodes... 2023-10-11T10:43:10.856 INFO:tasks.ceph:Setting up mgr nodes... 2023-10-11T10:43:10.856 DEBUG:teuthology.orchestra.run.smithi050:> sudo mkdir -p /var/lib/ceph/mgr/cluster2-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/cluster2-x/keyring 2023-10-11T10:43:10.921 INFO:teuthology.orchestra.run.smithi050.stdout:creating /var/lib/ceph/mgr/cluster2-x/keyring 2023-10-11T10:43:10.924 INFO:tasks.ceph:Setting up mds nodes... 2023-10-11T10:43:10.924 INFO:tasks.ceph_client:Setting up client nodes... 2023-10-11T10:43:10.925 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/cluster2.client.0.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.0.keyring 2023-10-11T10:43:10.972 INFO:teuthology.orchestra.run.smithi050.stdout:creating /etc/ceph/cluster2.client.0.keyring 2023-10-11T10:43:10.986 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror /etc/ceph/cluster2.client.mirror.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.keyring 2023-10-11T10:43:11.029 INFO:teuthology.orchestra.run.smithi111.stdout:creating /etc/ceph/cluster2.client.mirror.keyring 2023-10-11T10:43:11.042 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.0 /etc/ceph/cluster2.client.mirror.0.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.0.keyring 2023-10-11T10:43:11.125 INFO:teuthology.orchestra.run.smithi111.stdout:creating /etc/ceph/cluster2.client.mirror.0.keyring 2023-10-11T10:43:11.138 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.1 /etc/ceph/cluster2.client.mirror.1.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.1.keyring 2023-10-11T10:43:11.221 INFO:teuthology.orchestra.run.smithi111.stdout:creating /etc/ceph/cluster2.client.mirror.1.keyring 2023-10-11T10:43:11.234 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.2 /etc/ceph/cluster2.client.mirror.2.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.2.keyring 2023-10-11T10:43:11.319 INFO:teuthology.orchestra.run.smithi111.stdout:creating /etc/ceph/cluster2.client.mirror.2.keyring 2023-10-11T10:43:11.331 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.3 /etc/ceph/cluster2.client.mirror.3.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.3.keyring 2023-10-11T10:43:11.374 INFO:teuthology.orchestra.run.smithi111.stdout:creating /etc/ceph/cluster2.client.mirror.3.keyring 2023-10-11T10:43:11.386 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.4 /etc/ceph/cluster2.client.mirror.4.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.4.keyring 2023-10-11T10:43:11.469 INFO:teuthology.orchestra.run.smithi111.stdout:creating /etc/ceph/cluster2.client.mirror.4.keyring 2023-10-11T10:43:11.482 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.5 /etc/ceph/cluster2.client.mirror.5.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.5.keyring 2023-10-11T10:43:11.565 INFO:teuthology.orchestra.run.smithi111.stdout:creating /etc/ceph/cluster2.client.mirror.5.keyring 2023-10-11T10:43:11.578 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.6 /etc/ceph/cluster2.client.mirror.6.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.6.keyring 2023-10-11T10:43:11.661 INFO:teuthology.orchestra.run.smithi111.stdout:creating /etc/ceph/cluster2.client.mirror.6.keyring 2023-10-11T10:43:11.673 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-10-11T10:43:11.674 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi050.front.sepia.ceph.com'): {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi111.front.sepia.ceph.com'): {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'}} 2023-10-11T10:43:11.674 DEBUG:teuthology.orchestra.run.smithi111:> sudo mkdir -p /var/lib/ceph/osd/cluster2-0 2023-10-11T10:43:11.739 INFO:tasks.ceph:roles_to_devs: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2023-10-11T10:43:11.739 INFO:tasks.ceph:role: cluster2.osd.0 2023-10-11T10:43:11.739 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi111.front.sepia.ceph.com 2023-10-11T10:43:11.739 DEBUG:teuthology.orchestra.run.smithi111:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-10-11T10:43:11.812 INFO:teuthology.orchestra.run.smithi111.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-10-11T10:43:11.812 INFO:teuthology.orchestra.run.smithi111.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-11T10:43:11.813 INFO:teuthology.orchestra.run.smithi111.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-11T10:43:11.813 INFO:teuthology.orchestra.run.smithi111.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-11T10:43:11.813 INFO:teuthology.orchestra.run.smithi111.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-11T10:43:11.813 INFO:teuthology.orchestra.run.smithi111.stdout: = sunit=0 swidth=0 blks 2023-10-11T10:43:11.813 INFO:teuthology.orchestra.run.smithi111.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-11T10:43:11.814 INFO:teuthology.orchestra.run.smithi111.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-11T10:43:11.814 INFO:teuthology.orchestra.run.smithi111.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-11T10:43:11.814 INFO:teuthology.orchestra.run.smithi111.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-11T10:43:12.239 INFO:teuthology.orchestra.run.smithi111.stdout:Discarding blocks...Done. 2023-10-11T10:43:12.240 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi111.front.sepia.ceph.com -o noatime 2023-10-11T10:43:12.241 DEBUG:teuthology.orchestra.run.smithi111:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/cluster2-0 2023-10-11T10:43:12.419 DEBUG:teuthology.orchestra.run.smithi111:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster2-0 2023-10-11T10:43:12.450 DEBUG:teuthology.orchestra.run.smithi111:> sudo mkdir -p /var/lib/ceph/osd/cluster2-1 2023-10-11T10:43:12.516 INFO:tasks.ceph:roles_to_devs: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2023-10-11T10:43:12.516 INFO:tasks.ceph:role: cluster2.osd.1 2023-10-11T10:43:12.516 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi111.front.sepia.ceph.com 2023-10-11T10:43:12.517 DEBUG:teuthology.orchestra.run.smithi111:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-10-11T10:43:12.590 INFO:teuthology.orchestra.run.smithi111.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-10-11T10:43:12.590 INFO:teuthology.orchestra.run.smithi111.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-11T10:43:12.590 INFO:teuthology.orchestra.run.smithi111.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-11T10:43:12.590 INFO:teuthology.orchestra.run.smithi111.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-11T10:43:12.590 INFO:teuthology.orchestra.run.smithi111.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-11T10:43:12.591 INFO:teuthology.orchestra.run.smithi111.stdout: = sunit=0 swidth=0 blks 2023-10-11T10:43:12.591 INFO:teuthology.orchestra.run.smithi111.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-11T10:43:12.591 INFO:teuthology.orchestra.run.smithi111.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-11T10:43:12.591 INFO:teuthology.orchestra.run.smithi111.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-11T10:43:12.591 INFO:teuthology.orchestra.run.smithi111.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-11T10:43:13.010 INFO:teuthology.orchestra.run.smithi111.stdout:Discarding blocks...Done. 2023-10-11T10:43:13.012 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi111.front.sepia.ceph.com -o noatime 2023-10-11T10:43:13.012 DEBUG:teuthology.orchestra.run.smithi111:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/cluster2-1 2023-10-11T10:43:13.112 DEBUG:teuthology.orchestra.run.smithi111:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster2-1 2023-10-11T10:43:13.142 DEBUG:teuthology.orchestra.run.smithi111:> sudo mkdir -p /var/lib/ceph/osd/cluster2-2 2023-10-11T10:43:13.216 INFO:tasks.ceph:roles_to_devs: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2023-10-11T10:43:13.216 INFO:tasks.ceph:role: cluster2.osd.2 2023-10-11T10:43:13.216 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi111.front.sepia.ceph.com 2023-10-11T10:43:13.217 DEBUG:teuthology.orchestra.run.smithi111:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-10-11T10:43:13.302 INFO:teuthology.orchestra.run.smithi111.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-10-11T10:43:13.302 INFO:teuthology.orchestra.run.smithi111.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-11T10:43:13.303 INFO:teuthology.orchestra.run.smithi111.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-11T10:43:13.303 INFO:teuthology.orchestra.run.smithi111.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-11T10:43:13.303 INFO:teuthology.orchestra.run.smithi111.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-11T10:43:13.303 INFO:teuthology.orchestra.run.smithi111.stdout: = sunit=0 swidth=0 blks 2023-10-11T10:43:13.303 INFO:teuthology.orchestra.run.smithi111.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-11T10:43:13.304 INFO:teuthology.orchestra.run.smithi111.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-11T10:43:13.304 INFO:teuthology.orchestra.run.smithi111.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-11T10:43:13.304 INFO:teuthology.orchestra.run.smithi111.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-11T10:43:13.744 INFO:teuthology.orchestra.run.smithi111.stdout:Discarding blocks...Done. 2023-10-11T10:43:13.745 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi111.front.sepia.ceph.com -o noatime 2023-10-11T10:43:13.746 DEBUG:teuthology.orchestra.run.smithi111:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/cluster2-2 2023-10-11T10:43:13.892 DEBUG:teuthology.orchestra.run.smithi111:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster2-2 2023-10-11T10:43:13.923 DEBUG:teuthology.orchestra.run.smithi111:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster2 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/cluster2.monmap 2023-10-11T10:43:14.033 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T10:43:14.032+0000 7f3c441ba640 -1 auth: error reading file: /var/lib/ceph/osd/cluster2-0/keyring: can't open /var/lib/ceph/osd/cluster2-0/keyring: (2) No such file or directory 2023-10-11T10:43:14.036 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T10:43:14.035+0000 7f3c441ba640 -1 created new key in keyring /var/lib/ceph/osd/cluster2-0/keyring 2023-10-11T10:43:14.036 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T10:43:14.035+0000 7f3c441ba640 -1 bluestore(/var/lib/ceph/osd/cluster2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-0/block: (2) No such file or directory 2023-10-11T10:43:14.036 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T10:43:14.035+0000 7f3c441ba640 -1 bluestore(/var/lib/ceph/osd/cluster2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-0/block: (2) No such file or directory 2023-10-11T10:43:14.036 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T10:43:14.035+0000 7f3c441ba640 -1 bluestore(/var/lib/ceph/osd/cluster2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-0/block: (2) No such file or directory 2023-10-11T10:43:14.040 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T10:43:14.040+0000 7f3c441ba640 -1 bluestore(/var/lib/ceph/osd/cluster2-0) _read_fsid unparsable uuid 2023-10-11T10:43:16.904 DEBUG:teuthology.orchestra.run.smithi111:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster2-0 2023-10-11T10:43:16.942 DEBUG:teuthology.orchestra.run.smithi111:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster2 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/cluster2.monmap 2023-10-11T10:43:17.034 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T10:43:17.033+0000 7fcca32c7640 -1 auth: error reading file: /var/lib/ceph/osd/cluster2-1/keyring: can't open /var/lib/ceph/osd/cluster2-1/keyring: (2) No such file or directory 2023-10-11T10:43:17.034 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T10:43:17.033+0000 7fcca32c7640 -1 created new key in keyring /var/lib/ceph/osd/cluster2-1/keyring 2023-10-11T10:43:17.034 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T10:43:17.033+0000 7fcca32c7640 -1 bluestore(/var/lib/ceph/osd/cluster2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-1/block: (2) No such file or directory 2023-10-11T10:43:17.035 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T10:43:17.033+0000 7fcca32c7640 -1 bluestore(/var/lib/ceph/osd/cluster2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-1/block: (2) No such file or directory 2023-10-11T10:43:17.035 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T10:43:17.033+0000 7fcca32c7640 -1 bluestore(/var/lib/ceph/osd/cluster2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-1/block: (2) No such file or directory 2023-10-11T10:43:17.035 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T10:43:17.033+0000 7fcca32c7640 -1 bluestore(/var/lib/ceph/osd/cluster2-1) _read_fsid unparsable uuid 2023-10-11T10:43:19.897 DEBUG:teuthology.orchestra.run.smithi111:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster2-1 2023-10-11T10:43:19.928 DEBUG:teuthology.orchestra.run.smithi111:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster2 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/cluster2.monmap 2023-10-11T10:43:20.019 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T10:43:20.018+0000 7f22730e9640 -1 auth: error reading file: /var/lib/ceph/osd/cluster2-2/keyring: can't open /var/lib/ceph/osd/cluster2-2/keyring: (2) No such file or directory 2023-10-11T10:43:20.020 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T10:43:20.018+0000 7f22730e9640 -1 created new key in keyring /var/lib/ceph/osd/cluster2-2/keyring 2023-10-11T10:43:20.020 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T10:43:20.018+0000 7f22730e9640 -1 bluestore(/var/lib/ceph/osd/cluster2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-2/block: (2) No such file or directory 2023-10-11T10:43:20.020 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T10:43:20.018+0000 7f22730e9640 -1 bluestore(/var/lib/ceph/osd/cluster2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-2/block: (2) No such file or directory 2023-10-11T10:43:20.021 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T10:43:20.019+0000 7f22730e9640 -1 bluestore(/var/lib/ceph/osd/cluster2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-2/block: (2) No such file or directory 2023-10-11T10:43:20.021 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T10:43:20.019+0000 7f22730e9640 -1 bluestore(/var/lib/ceph/osd/cluster2-2) _read_fsid unparsable uuid 2023-10-11T10:43:22.890 DEBUG:teuthology.orchestra.run.smithi111:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster2-2 2023-10-11T10:43:22.921 INFO:tasks.ceph:Reading keys from all nodes... 2023-10-11T10:43:22.921 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2023-10-11T10:43:22.922 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd if=/var/lib/ceph/mgr/cluster2-x/keyring of=/dev/stdout 2023-10-11T10:43:22.955 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2023-10-11T10:43:22.956 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd if=/var/lib/ceph/osd/cluster2-0/keyring of=/dev/stdout 2023-10-11T10:43:22.989 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2023-10-11T10:43:22.989 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd if=/var/lib/ceph/osd/cluster2-1/keyring of=/dev/stdout 2023-10-11T10:43:23.056 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2023-10-11T10:43:23.056 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd if=/var/lib/ceph/osd/cluster2-2/keyring of=/dev/stdout 2023-10-11T10:43:23.123 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2023-10-11T10:43:23.123 DEBUG:teuthology.orchestra.run.smithi050:> dd if=/etc/ceph/cluster2.client.0.keyring of=/dev/stdout 2023-10-11T10:43:23.144 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2023-10-11T10:43:23.144 DEBUG:teuthology.orchestra.run.smithi111:> dd if=/etc/ceph/cluster2.client.mirror.keyring of=/dev/stdout 2023-10-11T10:43:23.181 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2023-10-11T10:43:23.181 DEBUG:teuthology.orchestra.run.smithi111:> dd if=/etc/ceph/cluster2.client.mirror.0.keyring of=/dev/stdout 2023-10-11T10:43:23.238 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2023-10-11T10:43:23.238 DEBUG:teuthology.orchestra.run.smithi111:> dd if=/etc/ceph/cluster2.client.mirror.1.keyring of=/dev/stdout 2023-10-11T10:43:23.295 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2023-10-11T10:43:23.295 DEBUG:teuthology.orchestra.run.smithi111:> dd if=/etc/ceph/cluster2.client.mirror.2.keyring of=/dev/stdout 2023-10-11T10:43:23.351 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2023-10-11T10:43:23.352 DEBUG:teuthology.orchestra.run.smithi111:> dd if=/etc/ceph/cluster2.client.mirror.3.keyring of=/dev/stdout 2023-10-11T10:43:23.407 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2023-10-11T10:43:23.408 DEBUG:teuthology.orchestra.run.smithi111:> dd if=/etc/ceph/cluster2.client.mirror.4.keyring of=/dev/stdout 2023-10-11T10:43:23.464 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2023-10-11T10:43:23.464 DEBUG:teuthology.orchestra.run.smithi111:> dd if=/etc/ceph/cluster2.client.mirror.5.keyring of=/dev/stdout 2023-10-11T10:43:23.521 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2023-10-11T10:43:23.521 DEBUG:teuthology.orchestra.run.smithi111:> dd if=/etc/ceph/cluster2.client.mirror.6.keyring of=/dev/stdout 2023-10-11T10:43:23.578 INFO:tasks.ceph:Adding keys to all mons... 2023-10-11T10:43:23.578 DEBUG:teuthology.orchestra.run.smithi111:> sudo tee -a /etc/ceph/cluster2.keyring 2023-10-11T10:43:23.643 INFO:teuthology.orchestra.run.smithi111.stdout:[mgr.x] 2023-10-11T10:43:23.643 INFO:teuthology.orchestra.run.smithi111.stdout: key = AQA+fCZlaRP0NhAAp+WlN3L0g/5a9KywbuyhlQ== 2023-10-11T10:43:23.643 INFO:teuthology.orchestra.run.smithi111.stdout:[osd.0] 2023-10-11T10:43:23.643 INFO:teuthology.orchestra.run.smithi111.stdout: key = AQBCfCZlsh0DAhAAa2UI5Q05n3Ia6EeqiWlaaA== 2023-10-11T10:43:23.644 INFO:teuthology.orchestra.run.smithi111.stdout:[osd.1] 2023-10-11T10:43:23.644 INFO:teuthology.orchestra.run.smithi111.stdout: key = AQBFfCZlTeMGAhAAD2tXH5HAzmFtuXM4SvgpDQ== 2023-10-11T10:43:23.644 INFO:teuthology.orchestra.run.smithi111.stdout:[osd.2] 2023-10-11T10:43:23.644 INFO:teuthology.orchestra.run.smithi111.stdout: key = AQBIfCZlO0ssARAAXoTb5JxGV9TKMv+53XEnDw== 2023-10-11T10:43:23.644 INFO:teuthology.orchestra.run.smithi111.stdout:[client.0] 2023-10-11T10:43:23.644 INFO:teuthology.orchestra.run.smithi111.stdout: key = AQA+fCZlaEUBOhAAro9KCpP4+rrWxoN+u+is5Q== 2023-10-11T10:43:23.645 INFO:teuthology.orchestra.run.smithi111.stdout:[client.mirror] 2023-10-11T10:43:23.645 INFO:teuthology.orchestra.run.smithi111.stdout: key = AQA/fCZlh3vSARAAikuq2YV3q8iwr5PlIm8NtQ== 2023-10-11T10:43:23.645 INFO:teuthology.orchestra.run.smithi111.stdout:[client.mirror.0] 2023-10-11T10:43:23.645 INFO:teuthology.orchestra.run.smithi111.stdout: key = AQA/fCZlrLKFBxAAtIcmebwDjJxnvs+6zwY+nA== 2023-10-11T10:43:23.645 INFO:teuthology.orchestra.run.smithi111.stdout:[client.mirror.1] 2023-10-11T10:43:23.646 INFO:teuthology.orchestra.run.smithi111.stdout: key = AQA/fCZl9F87DRAAxwv9S5pnPwjuVLpzMmjyeA== 2023-10-11T10:43:23.646 INFO:teuthology.orchestra.run.smithi111.stdout:[client.mirror.2] 2023-10-11T10:43:23.646 INFO:teuthology.orchestra.run.smithi111.stdout: key = AQA/fCZlP7UNExAAR7EYAsG+WkpNVC51Pve+zg== 2023-10-11T10:43:23.646 INFO:teuthology.orchestra.run.smithi111.stdout:[client.mirror.3] 2023-10-11T10:43:23.646 INFO:teuthology.orchestra.run.smithi111.stdout: key = AQA/fCZlRbRVFhAABrKtvCT9iJoU7LA1u3kEsA== 2023-10-11T10:43:23.646 INFO:teuthology.orchestra.run.smithi111.stdout:[client.mirror.4] 2023-10-11T10:43:23.647 INFO:teuthology.orchestra.run.smithi111.stdout: key = AQA/fCZlXSgEHBAAT2JCxJRSJCDVd4j4NJsi8g== 2023-10-11T10:43:23.647 INFO:teuthology.orchestra.run.smithi111.stdout:[client.mirror.5] 2023-10-11T10:43:23.647 INFO:teuthology.orchestra.run.smithi111.stdout: key = AQA/fCZlXHu6IRAA6hv48TOf11N1wGDMrPjbhw== 2023-10-11T10:43:23.647 INFO:teuthology.orchestra.run.smithi111.stdout:[client.mirror.6] 2023-10-11T10:43:23.647 INFO:teuthology.orchestra.run.smithi111.stdout: key = AQA/fCZlTxVyJxAAY/Py23+pJrwvVCWEIbE4tw== 2023-10-11T10:43:23.648 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-10-11T10:43:23.741 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-11T10:43:23.794 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-11T10:43:23.846 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-11T10:43:23.898 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-11T10:43:23.953 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-11T10:43:24.009 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-11T10:43:24.064 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-11T10:43:24.121 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-11T10:43:24.176 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-11T10:43:24.232 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.4 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-11T10:43:24.287 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.5 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-11T10:43:24.342 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.6 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-11T10:43:24.398 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-10-11T10:43:24.399 DEBUG:teuthology.orchestra.run.smithi111:> sudo mkdir -p /var/lib/ceph/mon/cluster2-a 2023-10-11T10:43:24.425 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster cluster2 --mkfs -i a --monmap /home/ubuntu/cephtest/cluster2.monmap --keyring /etc/ceph/cluster2.keyring 2023-10-11T10:43:24.523 DEBUG:teuthology.orchestra.run.smithi111:> sudo chown -R ceph:ceph /var/lib/ceph/mon/cluster2-a 2023-10-11T10:43:24.549 DEBUG:teuthology.orchestra.run.smithi111:> rm -- /home/ubuntu/cephtest/cluster2.monmap 2023-10-11T10:43:24.606 INFO:tasks.ceph:Starting mon daemons in cluster cluster2... 2023-10-11T10:43:24.607 INFO:tasks.ceph.cluster2.mon.a:Restarting daemon 2023-10-11T10:43:24.607 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster cluster2 -i a 2023-10-11T10:43:24.649 INFO:tasks.ceph.cluster2.mon.a:Started 2023-10-11T10:43:24.650 INFO:tasks.ceph:Starting mgr daemons in cluster cluster2... 2023-10-11T10:43:24.650 INFO:tasks.ceph.cluster2.mgr.x:Restarting daemon 2023-10-11T10:43:24.650 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster cluster2 -i x 2023-10-11T10:43:24.653 INFO:tasks.ceph.cluster2.mgr.x:Started 2023-10-11T10:43:24.654 INFO:tasks.ceph:Setting crush tunables to default 2023-10-11T10:43:24.654 DEBUG:teuthology.orchestra.run.smithi111:> sudo ceph --cluster cluster2 osd crush tunables default 2023-10-11T10:43:24.881 INFO:teuthology.orchestra.run.smithi111.stderr:adjusted tunables profile to default 2023-10-11T10:43:24.892 INFO:tasks.ceph:check_enable_crimson: False 2023-10-11T10:43:24.893 INFO:tasks.ceph:Starting osd daemons in cluster cluster2... 2023-10-11T10:43:24.893 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2023-10-11T10:43:24.893 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd if=/var/lib/ceph/osd/cluster2-0/fsid of=/dev/stdout 2023-10-11T10:43:24.903 INFO:tasks.ceph.cluster2.mgr.x.smithi050.stderr:2023-10-11T10:43:24.902+0000 7fa025aa2200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-11T10:43:24.919 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2023-10-11T10:43:24.920 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd if=/var/lib/ceph/osd/cluster2-1/fsid of=/dev/stdout 2023-10-11T10:43:24.971 INFO:tasks.ceph.cluster2.mgr.x.smithi050.stderr:2023-10-11T10:43:24.970+0000 7fa025aa2200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-11T10:43:24.985 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2023-10-11T10:43:24.986 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd if=/var/lib/ceph/osd/cluster2-2/fsid of=/dev/stdout 2023-10-11T10:43:25.044 INFO:tasks.ceph.cluster2.mgr.x.smithi050.stderr:2023-10-11T10:43:25.043+0000 7fa025aa2200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-11T10:43:25.051 DEBUG:teuthology.orchestra.run.smithi111:> sudo ceph --cluster cluster2 osd new c8da6d23-27fa-432b-be3d-69bdaf106fce 0 2023-10-11T10:43:25.125 INFO:tasks.ceph.cluster2.mgr.x.smithi050.stderr:2023-10-11T10:43:25.124+0000 7fa025aa2200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-11T10:43:25.271 INFO:tasks.ceph.cluster2.mgr.x.smithi050.stderr:2023-10-11T10:43:25.270+0000 7fa025aa2200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-11T10:43:25.300 INFO:teuthology.orchestra.run.smithi111.stdout:0 2023-10-11T10:43:25.312 DEBUG:teuthology.orchestra.run.smithi111:> sudo ceph --cluster cluster2 osd new 50f479e5-1ac9-46fc-8007-7c47997d0298 1 2023-10-11T10:43:25.518 INFO:teuthology.orchestra.run.smithi111.stdout:1 2023-10-11T10:43:25.531 DEBUG:teuthology.orchestra.run.smithi111:> sudo ceph --cluster cluster2 osd new df3b4e09-205a-4927-abf2-2c18e5f1cdeb 2 2023-10-11T10:43:25.609 INFO:tasks.ceph.cluster2.mgr.x.smithi050.stderr:2023-10-11T10:43:25.608+0000 7fa025aa2200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-11T10:43:25.728 INFO:tasks.ceph.cluster2.mgr.x.smithi050.stderr:2023-10-11T10:43:25.727+0000 7fa025aa2200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-11T10:43:25.736 INFO:teuthology.orchestra.run.smithi111.stdout:2 2023-10-11T10:43:25.749 INFO:tasks.ceph.cluster2.osd.0:Restarting daemon 2023-10-11T10:43:25.749 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster2 -i 0 2023-10-11T10:43:25.751 INFO:tasks.ceph.cluster2.osd.0:Started 2023-10-11T10:43:25.752 INFO:tasks.ceph.cluster2.osd.1:Restarting daemon 2023-10-11T10:43:25.752 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster2 -i 1 2023-10-11T10:43:25.754 INFO:tasks.ceph.cluster2.osd.1:Started 2023-10-11T10:43:25.754 INFO:tasks.ceph.cluster2.osd.2:Restarting daemon 2023-10-11T10:43:25.755 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster2 -i 2 2023-10-11T10:43:25.757 INFO:tasks.ceph.cluster2.osd.2:Started 2023-10-11T10:43:25.757 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2023-10-11T10:43:25.804 INFO:tasks.ceph.cluster2.mgr.x.smithi050.stderr:2023-10-11T10:43:25.803+0000 7fa025aa2200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-11T10:43:25.887 INFO:tasks.ceph.cluster2.mgr.x.smithi050.stderr:2023-10-11T10:43:25.886+0000 7fa025aa2200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-11T10:43:25.964 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T10:43:25.964 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":5,"fsid":"022ce6f2-9a5b-45ad-8b77-a1332d92f228","created":"2023-10-11T10:43:24.777178+0000","modified":"2023-10-11T10:43:25.735756+0000","last_up_change":"0.000000","last_in_change":"2023-10-11T10:43:25.735756+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c8da6d23-27fa-432b-be3d-69bdaf106fce","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"50f479e5-1ac9-46fc-8007-7c47997d0298","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"df3b4e09-205a-4927-abf2-2c18e5f1cdeb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-11T10:43:25.976 INFO:tasks.ceph.ceph_manager.cluster2:[] 2023-10-11T10:43:25.976 INFO:tasks.ceph:Waiting for OSDs to come up 2023-10-11T10:43:26.270 INFO:tasks.ceph.cluster2.mgr.x.smithi050.stderr:2023-10-11T10:43:26.269+0000 7fa025aa2200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-11T10:43:26.279 DEBUG:teuthology.orchestra.run.smithi111:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster2 osd dump --format=json 2023-10-11T10:43:26.364 INFO:tasks.ceph.cluster2.osd.0.smithi111.stderr:2023-10-11T10:43:26.363+0000 7f80d80a7640 -1 Falling back to public interface 2023-10-11T10:43:26.368 INFO:tasks.ceph.cluster2.osd.1.smithi111.stderr:2023-10-11T10:43:26.367+0000 7f9641e52640 -1 Falling back to public interface 2023-10-11T10:43:26.368 INFO:tasks.ceph.cluster2.osd.2.smithi111.stderr:2023-10-11T10:43:26.367+0000 7fa3754d9640 -1 Falling back to public interface 2023-10-11T10:43:26.369 INFO:tasks.ceph.cluster2.osd.1.smithi111.stderr:2023-10-11T10:43:26.368+0000 7f9639210700 -1 PosixStack listen unable to listen on v1:0.0.0.0:6809/0: (98) Address already in use 2023-10-11T10:43:26.468 INFO:teuthology.misc.health.smithi111.stdout: 2023-10-11T10:43:26.469 INFO:teuthology.misc.health.smithi111.stdout:{"epoch":5,"fsid":"022ce6f2-9a5b-45ad-8b77-a1332d92f228","created":"2023-10-11T10:43:24.777178+0000","modified":"2023-10-11T10:43:25.735756+0000","last_up_change":"0.000000","last_in_change":"2023-10-11T10:43:25.735756+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c8da6d23-27fa-432b-be3d-69bdaf106fce","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"50f479e5-1ac9-46fc-8007-7c47997d0298","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"df3b4e09-205a-4927-abf2-2c18e5f1cdeb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-11T10:43:26.479 DEBUG:teuthology.misc:0 of 3 OSDs are up 2023-10-11T10:43:26.875 INFO:tasks.ceph.cluster2.mgr.x.smithi050.stderr:2023-10-11T10:43:26.875+0000 7fa025aa2200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-11T10:43:26.947 INFO:tasks.ceph.cluster2.mgr.x.smithi050.stderr:2023-10-11T10:43:26.946+0000 7fa025aa2200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-11T10:43:27.604 INFO:tasks.ceph.cluster2.mgr.x.smithi050.stderr:2023-10-11T10:43:27.604+0000 7fa025aa2200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-11T10:43:27.683 INFO:tasks.ceph.cluster2.mgr.x.smithi050.stderr:2023-10-11T10:43:27.683+0000 7fa025aa2200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-11T10:43:27.832 INFO:tasks.ceph.cluster2.mgr.x.smithi050.stderr:2023-10-11T10:43:27.832+0000 7fa025aa2200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-11T10:43:28.064 INFO:tasks.ceph.cluster2.mgr.x.smithi050.stderr:2023-10-11T10:43:28.064+0000 7fa025aa2200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-11T10:43:28.140 INFO:tasks.ceph.cluster2.mgr.x.smithi050.stderr:2023-10-11T10:43:28.140+0000 7fa025aa2200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-11T10:43:28.415 INFO:tasks.ceph.cluster2.mgr.x.smithi050.stderr:2023-10-11T10:43:28.415+0000 7fa025aa2200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-11T10:43:28.638 INFO:tasks.ceph.cluster2.mgr.x.smithi050.stderr:2023-10-11T10:43:28.638+0000 7fa025aa2200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-11T10:43:28.790 INFO:tasks.ceph.cluster2.mgr.x.smithi050.stderr:2023-10-11T10:43:28.790+0000 7fa025aa2200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-11T10:43:28.907 INFO:tasks.ceph.cluster2.mgr.x.smithi050.stderr:2023-10-11T10:43:28.907+0000 7fa025aa2200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-11T10:43:29.065 INFO:tasks.ceph.cluster2.mgr.x.smithi050.stderr:2023-10-11T10:43:29.065+0000 7fa025aa2200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-11T10:43:29.336 INFO:tasks.ceph.cluster2.mgr.x.smithi050.stderr:2023-10-11T10:43:29.336+0000 7fa025aa2200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-11T10:43:29.498 INFO:tasks.ceph.cluster2.mgr.x.smithi050.stderr:2023-10-11T10:43:29.498+0000 7fa025aa2200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-11T10:43:29.838 INFO:tasks.ceph.cluster2.osd.0.smithi111.stderr:2023-10-11T10:43:29.838+0000 7f80d80a7640 -1 osd.0 0 log_to_monitors true 2023-10-11T10:43:29.899 INFO:tasks.ceph.cluster2.osd.1.smithi111.stderr:2023-10-11T10:43:29.899+0000 7f9641e52640 -1 osd.1 0 log_to_monitors true 2023-10-11T10:43:29.906 INFO:tasks.ceph.cluster2.osd.2.smithi111.stderr:2023-10-11T10:43:29.905+0000 7fa3754d9640 -1 osd.2 0 log_to_monitors true 2023-10-11T10:43:30.041 INFO:tasks.ceph.cluster2.mgr.x.smithi050.stderr:2023-10-11T10:43:30.041+0000 7fa025aa2200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-11T10:43:31.262 INFO:tasks.ceph.cluster2.mgr.x.smithi050.stderr:2023-10-11T10:43:31.262+0000 7f9fdda4c700 -1 mgr.server handle_report got status from non-daemon mon.a 2023-10-11T10:43:32.783 DEBUG:teuthology.orchestra.run.smithi111:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster2 osd dump --format=json 2023-10-11T10:43:33.249 INFO:teuthology.misc.health.smithi111.stdout: 2023-10-11T10:43:33.250 INFO:teuthology.misc.health.smithi111.stdout:{"epoch":7,"fsid":"022ce6f2-9a5b-45ad-8b77-a1332d92f228","created":"2023-10-11T10:43:24.777178+0000","modified":"2023-10-11T10:43:31.739492+0000","last_up_change":"0.000000","last_in_change":"2023-10-11T10:43:25.735756+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c8da6d23-27fa-432b-be3d-69bdaf106fce","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"50f479e5-1ac9-46fc-8007-7c47997d0298","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"df3b4e09-205a-4927-abf2-2c18e5f1cdeb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-11T10:43:33.266 DEBUG:teuthology.misc:0 of 3 OSDs are up 2023-10-11T10:43:33.412 INFO:tasks.ceph.cluster2.osd.0.smithi111.stderr:2023-10-11T10:43:33.409+0000 7f80cbc5e700 -1 osd.0 0 waiting for initial osdmap 2023-10-11T10:43:33.431 INFO:tasks.ceph.cluster2.osd.0.smithi111.stderr:2023-10-11T10:43:33.431+0000 7f80c4039700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-11T10:43:33.490 INFO:tasks.ceph.cluster2.osd.1.smithi111.stderr:2023-10-11T10:43:33.490+0000 7f9635a09700 -1 osd.1 0 waiting for initial osdmap 2023-10-11T10:43:33.503 INFO:tasks.ceph.cluster2.osd.1.smithi111.stderr:2023-10-11T10:43:33.503+0000 7f962dde4700 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-11T10:43:33.826 INFO:tasks.ceph.cluster2.osd.2.smithi111.stderr:2023-10-11T10:43:33.826+0000 7fa369090700 -1 osd.2 0 waiting for initial osdmap 2023-10-11T10:43:33.849 INFO:tasks.ceph.cluster2.osd.2.smithi111.stderr:2023-10-11T10:43:33.849+0000 7fa36146b700 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-11T10:43:39.569 DEBUG:teuthology.orchestra.run.smithi111:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster2 osd dump --format=json 2023-10-11T10:43:39.892 INFO:teuthology.misc.health.smithi111.stdout: 2023-10-11T10:43:39.893 INFO:teuthology.misc.health.smithi111.stdout:{"epoch":12,"fsid":"022ce6f2-9a5b-45ad-8b77-a1332d92f228","created":"2023-10-11T10:43:24.777178+0000","modified":"2023-10-11T10:43:38.746221+0000","last_up_change":"2023-10-11T10:43:34.412171+0000","last_in_change":"2023-10-11T10:43:25.735756+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-11T10:43:36.101586+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c8da6d23-27fa-432b-be3d-69bdaf106fce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":2734416346},{"type":"v1","addr":"172.21.15.111:6801","nonce":2734416346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":2734416346},{"type":"v1","addr":"172.21.15.111:6803","nonce":2734416346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":2734416346},{"type":"v1","addr":"172.21.15.111:6807","nonce":2734416346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":2734416346},{"type":"v1","addr":"172.21.15.111:6805","nonce":2734416346}]},"public_addr":"172.21.15.111:6801/2734416346","cluster_addr":"172.21.15.111:6803/2734416346","heartbeat_back_addr":"172.21.15.111:6807/2734416346","heartbeat_front_addr":"172.21.15.111:6805/2734416346","state":["exists","up"]},{"osd":1,"uuid":"50f479e5-1ac9-46fc-8007-7c47997d0298","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":1083998740},{"type":"v1","addr":"172.21.15.111:6810","nonce":1083998740}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":1083998740},{"type":"v1","addr":"172.21.15.111:6815","nonce":1083998740}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":1083998740},{"type":"v1","addr":"172.21.15.111:6823","nonce":1083998740}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6817","nonce":1083998740},{"type":"v1","addr":"172.21.15.111:6820","nonce":1083998740}]},"public_addr":"172.21.15.111:6810/1083998740","cluster_addr":"172.21.15.111:6815/1083998740","heartbeat_back_addr":"172.21.15.111:6823/1083998740","heartbeat_front_addr":"172.21.15.111:6820/1083998740","state":["exists","up"]},{"osd":2,"uuid":"df3b4e09-205a-4927-abf2-2c18e5f1cdeb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6809","nonce":1857953398},{"type":"v1","addr":"172.21.15.111:6811","nonce":1857953398}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6813","nonce":1857953398},{"type":"v1","addr":"172.21.15.111:6814","nonce":1857953398}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6819","nonce":1857953398},{"type":"v1","addr":"172.21.15.111:6821","nonce":1857953398}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":1857953398},{"type":"v1","addr":"172.21.15.111:6818","nonce":1857953398}]},"public_addr":"172.21.15.111:6811/1857953398","cluster_addr":"172.21.15.111:6814/1857953398","heartbeat_back_addr":"172.21.15.111:6821/1857953398","heartbeat_front_addr":"172.21.15.111:6818/1857953398","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T10:43:30.883150+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T10:43:30.876024+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T10:43:30.878261+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-11T10:43:39.905 DEBUG:teuthology.misc:3 of 3 OSDs are up 2023-10-11T10:43:39.906 INFO:tasks.ceph:Creating RBD pool 2023-10-11T10:43:39.906 DEBUG:teuthology.orchestra.run.smithi111:> sudo ceph --cluster cluster2 osd pool create rbd 8 2023-10-11T10:43:40.759 INFO:teuthology.orchestra.run.smithi111.stderr:pool 'rbd' created 2023-10-11T10:43:40.773 DEBUG:teuthology.orchestra.run.smithi111:> sudo ceph --cluster cluster2 osd pool application enable rbd rbd --yes-i-really-mean-it 2023-10-11T10:43:41.763 INFO:teuthology.orchestra.run.smithi111.stderr:enabled application 'rbd' on pool 'rbd' 2023-10-11T10:43:41.778 INFO:tasks.ceph:Starting mds daemons in cluster cluster2... 2023-10-11T10:43:41.779 INFO:tasks.ceph:Waiting until cluster2 daemons up and pgs clean... 2023-10-11T10:43:41.779 INFO:tasks.ceph.ceph_manager.cluster2:waiting for mgr available 2023-10-11T10:43:41.780 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 mgr dump --format=json 2023-10-11T10:43:41.780 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-10-11T10:43:42.161 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T10:43:42.173 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":5,"active_gid":4100,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6826","nonce":620489281},{"type":"v1","addr":"172.21.15.50:6827","nonce":620489281}]},"active_addr":"172.21.15.50:6827/620489281","active_change":"2023-10-11T10:43:30.054577+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:0.0.3","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.50:0","nonce":1873987426}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.50:0","nonce":3884097886}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.50:0","nonce":3189561754}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.50:0","nonce":1923478968}]}]} 2023-10-11T10:43:42.175 INFO:tasks.ceph.ceph_manager.cluster2:mgr available! 2023-10-11T10:43:42.176 INFO:tasks.ceph.ceph_manager.cluster2:waiting for all up 2023-10-11T10:43:42.176 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2023-10-11T10:43:42.494 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T10:43:42.494 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":14,"fsid":"022ce6f2-9a5b-45ad-8b77-a1332d92f228","created":"2023-10-11T10:43:24.777178+0000","modified":"2023-10-11T10:43:41.755050+0000","last_up_change":"2023-10-11T10:43:34.412171+0000","last_in_change":"2023-10-11T10:43:25.735756+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-11T10:43:36.101586+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-11T10:43:40.242633+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c8da6d23-27fa-432b-be3d-69bdaf106fce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":2734416346},{"type":"v1","addr":"172.21.15.111:6801","nonce":2734416346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":2734416346},{"type":"v1","addr":"172.21.15.111:6803","nonce":2734416346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":2734416346},{"type":"v1","addr":"172.21.15.111:6807","nonce":2734416346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":2734416346},{"type":"v1","addr":"172.21.15.111:6805","nonce":2734416346}]},"public_addr":"172.21.15.111:6801/2734416346","cluster_addr":"172.21.15.111:6803/2734416346","heartbeat_back_addr":"172.21.15.111:6807/2734416346","heartbeat_front_addr":"172.21.15.111:6805/2734416346","state":["exists","up"]},{"osd":1,"uuid":"50f479e5-1ac9-46fc-8007-7c47997d0298","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":1083998740},{"type":"v1","addr":"172.21.15.111:6810","nonce":1083998740}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":1083998740},{"type":"v1","addr":"172.21.15.111:6815","nonce":1083998740}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":1083998740},{"type":"v1","addr":"172.21.15.111:6823","nonce":1083998740}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6817","nonce":1083998740},{"type":"v1","addr":"172.21.15.111:6820","nonce":1083998740}]},"public_addr":"172.21.15.111:6810/1083998740","cluster_addr":"172.21.15.111:6815/1083998740","heartbeat_back_addr":"172.21.15.111:6823/1083998740","heartbeat_front_addr":"172.21.15.111:6820/1083998740","state":["exists","up"]},{"osd":2,"uuid":"df3b4e09-205a-4927-abf2-2c18e5f1cdeb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6809","nonce":1857953398},{"type":"v1","addr":"172.21.15.111:6811","nonce":1857953398}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6813","nonce":1857953398},{"type":"v1","addr":"172.21.15.111:6814","nonce":1857953398}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6819","nonce":1857953398},{"type":"v1","addr":"172.21.15.111:6821","nonce":1857953398}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":1857953398},{"type":"v1","addr":"172.21.15.111:6818","nonce":1857953398}]},"public_addr":"172.21.15.111:6811/1857953398","cluster_addr":"172.21.15.111:6814/1857953398","heartbeat_back_addr":"172.21.15.111:6821/1857953398","heartbeat_front_addr":"172.21.15.111:6818/1857953398","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T10:43:30.883150+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T10:43:30.876024+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T10:43:30.878261+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-11T10:43:42.507 INFO:tasks.ceph.ceph_manager.cluster2:all up! 2023-10-11T10:43:42.508 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2023-10-11T10:43:42.830 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T10:43:42.830 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":15,"fsid":"022ce6f2-9a5b-45ad-8b77-a1332d92f228","created":"2023-10-11T10:43:24.777178+0000","modified":"2023-10-11T10:43:42.758688+0000","last_up_change":"2023-10-11T10:43:34.412171+0000","last_in_change":"2023-10-11T10:43:25.735756+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-11T10:43:36.101586+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-11T10:43:40.242633+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c8da6d23-27fa-432b-be3d-69bdaf106fce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":2734416346},{"type":"v1","addr":"172.21.15.111:6801","nonce":2734416346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":2734416346},{"type":"v1","addr":"172.21.15.111:6803","nonce":2734416346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":2734416346},{"type":"v1","addr":"172.21.15.111:6807","nonce":2734416346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":2734416346},{"type":"v1","addr":"172.21.15.111:6805","nonce":2734416346}]},"public_addr":"172.21.15.111:6801/2734416346","cluster_addr":"172.21.15.111:6803/2734416346","heartbeat_back_addr":"172.21.15.111:6807/2734416346","heartbeat_front_addr":"172.21.15.111:6805/2734416346","state":["exists","up"]},{"osd":1,"uuid":"50f479e5-1ac9-46fc-8007-7c47997d0298","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":1083998740},{"type":"v1","addr":"172.21.15.111:6810","nonce":1083998740}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":1083998740},{"type":"v1","addr":"172.21.15.111:6815","nonce":1083998740}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":1083998740},{"type":"v1","addr":"172.21.15.111:6823","nonce":1083998740}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6817","nonce":1083998740},{"type":"v1","addr":"172.21.15.111:6820","nonce":1083998740}]},"public_addr":"172.21.15.111:6810/1083998740","cluster_addr":"172.21.15.111:6815/1083998740","heartbeat_back_addr":"172.21.15.111:6823/1083998740","heartbeat_front_addr":"172.21.15.111:6820/1083998740","state":["exists","up"]},{"osd":2,"uuid":"df3b4e09-205a-4927-abf2-2c18e5f1cdeb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6809","nonce":1857953398},{"type":"v1","addr":"172.21.15.111:6811","nonce":1857953398}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6813","nonce":1857953398},{"type":"v1","addr":"172.21.15.111:6814","nonce":1857953398}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6819","nonce":1857953398},{"type":"v1","addr":"172.21.15.111:6821","nonce":1857953398}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":1857953398},{"type":"v1","addr":"172.21.15.111:6818","nonce":1857953398}]},"public_addr":"172.21.15.111:6811/1857953398","cluster_addr":"172.21.15.111:6814/1857953398","heartbeat_back_addr":"172.21.15.111:6821/1857953398","heartbeat_front_addr":"172.21.15.111:6818/1857953398","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T10:43:30.883150+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T10:43:30.876024+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T10:43:30.878261+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-11T10:43:42.844 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.0 flush_pg_stats 2023-10-11T10:43:42.844 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.1 flush_pg_stats 2023-10-11T10:43:42.845 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.2 flush_pg_stats 2023-10-11T10:43:43.012 INFO:teuthology.orchestra.run.smithi111.stdout:34359738374 2023-10-11T10:43:43.012 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.0 2023-10-11T10:43:43.016 INFO:teuthology.orchestra.run.smithi111.stdout:34359738372 2023-10-11T10:43:43.016 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.2 2023-10-11T10:43:43.017 INFO:teuthology.orchestra.run.smithi111.stdout:34359738373 2023-10-11T10:43:43.018 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.1 2023-10-11T10:43:43.361 INFO:teuthology.orchestra.run.smithi111.stdout:34359738373 2023-10-11T10:43:43.380 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738374 got 34359738373 for osd.0 2023-10-11T10:43:43.422 INFO:teuthology.orchestra.run.smithi111.stdout:34359738372 2023-10-11T10:43:43.423 INFO:teuthology.orchestra.run.smithi111.stdout:34359738371 2023-10-11T10:43:43.437 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738373 got 34359738372 for osd.1 2023-10-11T10:43:43.438 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738372 got 34359738371 for osd.2 2023-10-11T10:43:44.381 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.0 2023-10-11T10:43:44.439 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.1 2023-10-11T10:43:44.439 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.2 2023-10-11T10:43:44.742 INFO:teuthology.orchestra.run.smithi111.stdout:34359738373 2023-10-11T10:43:44.757 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738374 got 34359738373 for osd.0 2023-10-11T10:43:44.781 INFO:teuthology.orchestra.run.smithi111.stdout:34359738373 2023-10-11T10:43:44.793 INFO:teuthology.orchestra.run.smithi111.stdout:34359738372 2023-10-11T10:43:44.795 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738373 got 34359738373 for osd.1 2023-10-11T10:43:44.795 DEBUG:teuthology.parallel:result is None 2023-10-11T10:43:44.807 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738372 got 34359738372 for osd.2 2023-10-11T10:43:44.808 DEBUG:teuthology.parallel:result is None 2023-10-11T10:43:45.758 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.0 2023-10-11T10:43:46.082 INFO:teuthology.orchestra.run.smithi111.stdout:34359738374 2023-10-11T10:43:46.096 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738374 got 34359738374 for osd.0 2023-10-11T10:43:46.097 DEBUG:teuthology.parallel:result is None 2023-10-11T10:43:46.097 INFO:tasks.ceph.ceph_manager.cluster2:waiting for clean 2023-10-11T10:43:46.097 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-10-11T10:43:46.413 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T10:43:46.416 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2023-10-11T10:43:46.427 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":19,"stamp":"2023-10-11T10:43:46.061418+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":17,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":82604,"kb_used_data":1544,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":283032916,"statfs":{"total":289910292480,"available":289825705984,"internally_reserved":0,"allocated":1581056,"data_stored":1292330,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4782,"internal_metadata":82963794},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001304"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-10-11T10:43:42.762455+0000","last_change":"2023-10-11T10:43:41.768684+0000","last_active":"2023-10-11T10:43:42.762455+0000","last_peered":"2023-10-11T10:43:42.762455+0000","last_clean":"2023-10-11T10:43:42.762455+0000","last_became_active":"2023-10-11T10:43:41.768411+0000","last_became_peered":"2023-10-11T10:43:41.768411+0000","last_unstale":"2023-10-11T10:43:42.762455+0000","last_undegraded":"2023-10-11T10:43:42.762455+0000","last_fullsized":"2023-10-11T10:43:42.762455+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T15:00:09.263514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-10-11T10:43:42.762411+0000","last_change":"2023-10-11T10:43:41.767613+0000","last_active":"2023-10-11T10:43:42.762411+0000","last_peered":"2023-10-11T10:43:42.762411+0000","last_clean":"2023-10-11T10:43:42.762411+0000","last_became_active":"2023-10-11T10:43:41.767245+0000","last_became_peered":"2023-10-11T10:43:41.767245+0000","last_unstale":"2023-10-11T10:43:42.762411+0000","last_undegraded":"2023-10-11T10:43:42.762411+0000","last_fullsized":"2023-10-11T10:43:42.762411+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T13:14:42.980263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-10-11T10:43:42.762390+0000","last_change":"2023-10-11T10:43:41.767559+0000","last_active":"2023-10-11T10:43:42.762390+0000","last_peered":"2023-10-11T10:43:42.762390+0000","last_clean":"2023-10-11T10:43:42.762390+0000","last_became_active":"2023-10-11T10:43:41.767182+0000","last_became_peered":"2023-10-11T10:43:41.767182+0000","last_unstale":"2023-10-11T10:43:42.762390+0000","last_undegraded":"2023-10-11T10:43:42.762390+0000","last_fullsized":"2023-10-11T10:43:42.762390+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T13:33:09.638474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-10-11T10:43:43.804208+0000","last_change":"2023-10-11T10:43:41.772745+0000","last_active":"2023-10-11T10:43:43.804208+0000","last_peered":"2023-10-11T10:43:43.804208+0000","last_clean":"2023-10-11T10:43:43.804208+0000","last_became_active":"2023-10-11T10:43:41.772430+0000","last_became_peered":"2023-10-11T10:43:41.772430+0000","last_unstale":"2023-10-11T10:43:43.804208+0000","last_undegraded":"2023-10-11T10:43:43.804208+0000","last_fullsized":"2023-10-11T10:43:43.804208+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T18:51:15.753466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":141,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-10-11T10:43:41.764255+0000","last_change":"2023-10-11T10:43:37.764157+0000","last_active":"2023-10-11T10:43:41.764255+0000","last_peered":"2023-10-11T10:43:41.764255+0000","last_clean":"2023-10-11T10:43:41.764255+0000","last_became_active":"2023-10-11T10:43:37.762984+0000","last_became_peered":"2023-10-11T10:43:37.762984+0000","last_unstale":"2023-10-11T10:43:41.764255+0000","last_undegraded":"2023-10-11T10:43:41.764255+0000","last_fullsized":"2023-10-11T10:43:41.764255+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:36.739637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:36.739637+0000","last_clean_scrub_stamp":"2023-10-11T10:43:36.739637+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T19:04:42.047608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-10-11T10:43:42.762332+0000","last_change":"2023-10-11T10:43:41.772615+0000","last_active":"2023-10-11T10:43:42.762332+0000","last_peered":"2023-10-11T10:43:42.762332+0000","last_clean":"2023-10-11T10:43:42.762332+0000","last_became_active":"2023-10-11T10:43:41.772290+0000","last_became_peered":"2023-10-11T10:43:41.772290+0000","last_unstale":"2023-10-11T10:43:42.762332+0000","last_undegraded":"2023-10-11T10:43:42.762332+0000","last_fullsized":"2023-10-11T10:43:42.762332+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:26:06.909963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-10-11T10:43:43.804198+0000","last_change":"2023-10-11T10:43:41.772781+0000","last_active":"2023-10-11T10:43:43.804198+0000","last_peered":"2023-10-11T10:43:43.804198+0000","last_clean":"2023-10-11T10:43:43.804198+0000","last_became_active":"2023-10-11T10:43:41.772443+0000","last_became_peered":"2023-10-11T10:43:41.772443+0000","last_unstale":"2023-10-11T10:43:43.804198+0000","last_undegraded":"2023-10-11T10:43:43.804198+0000","last_fullsized":"2023-10-11T10:43:43.804198+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T14:15:30.547284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-10-11T10:43:43.804276+0000","last_change":"2023-10-11T10:43:41.767526+0000","last_active":"2023-10-11T10:43:43.804276+0000","last_peered":"2023-10-11T10:43:43.804276+0000","last_clean":"2023-10-11T10:43:43.804276+0000","last_became_active":"2023-10-11T10:43:41.767215+0000","last_became_peered":"2023-10-11T10:43:41.767215+0000","last_unstale":"2023-10-11T10:43:43.804276+0000","last_undegraded":"2023-10-11T10:43:43.804276+0000","last_fullsized":"2023-10-11T10:43:43.804276+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T15:44:46.953920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-10-11T10:43:42.762350+0000","last_change":"2023-10-11T10:43:41.767601+0000","last_active":"2023-10-11T10:43:42.762350+0000","last_peered":"2023-10-11T10:43:42.762350+0000","last_clean":"2023-10-11T10:43:42.762350+0000","last_became_active":"2023-10-11T10:43:41.767268+0000","last_became_peered":"2023-10-11T10:43:41.767268+0000","last_unstale":"2023-10-11T10:43:42.762350+0000","last_undegraded":"2023-10-11T10:43:42.762350+0000","last_fullsized":"2023-10-11T10:43:42.762350+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T20:13:11.818897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738373,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27148,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344692,"statfs":{"total":96636764160,"available":96608964608,"internally_reserved":0,"allocated":131072,"data_stored":37198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.224}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27200000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738375,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27728,"kb_used_data":708,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344112,"statfs":{"total":96636764160,"available":96608370688,"internally_reserved":0,"allocated":724992,"data_stored":627566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23200000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27200000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738374,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27728,"kb_used_data":708,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344112,"statfs":{"total":96636764160,"available":96608370688,"internally_reserved":0,"allocated":724992,"data_stored":627566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.31900000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34300000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-11T10:43:46.428 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-10-11T10:43:46.741 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T10:43:46.744 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2023-10-11T10:43:46.755 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":19,"stamp":"2023-10-11T10:43:46.061418+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":17,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":82604,"kb_used_data":1544,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":283032916,"statfs":{"total":289910292480,"available":289825705984,"internally_reserved":0,"allocated":1581056,"data_stored":1292330,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4782,"internal_metadata":82963794},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001304"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-10-11T10:43:42.762455+0000","last_change":"2023-10-11T10:43:41.768684+0000","last_active":"2023-10-11T10:43:42.762455+0000","last_peered":"2023-10-11T10:43:42.762455+0000","last_clean":"2023-10-11T10:43:42.762455+0000","last_became_active":"2023-10-11T10:43:41.768411+0000","last_became_peered":"2023-10-11T10:43:41.768411+0000","last_unstale":"2023-10-11T10:43:42.762455+0000","last_undegraded":"2023-10-11T10:43:42.762455+0000","last_fullsized":"2023-10-11T10:43:42.762455+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T15:00:09.263514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-10-11T10:43:42.762411+0000","last_change":"2023-10-11T10:43:41.767613+0000","last_active":"2023-10-11T10:43:42.762411+0000","last_peered":"2023-10-11T10:43:42.762411+0000","last_clean":"2023-10-11T10:43:42.762411+0000","last_became_active":"2023-10-11T10:43:41.767245+0000","last_became_peered":"2023-10-11T10:43:41.767245+0000","last_unstale":"2023-10-11T10:43:42.762411+0000","last_undegraded":"2023-10-11T10:43:42.762411+0000","last_fullsized":"2023-10-11T10:43:42.762411+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T13:14:42.980263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-10-11T10:43:42.762390+0000","last_change":"2023-10-11T10:43:41.767559+0000","last_active":"2023-10-11T10:43:42.762390+0000","last_peered":"2023-10-11T10:43:42.762390+0000","last_clean":"2023-10-11T10:43:42.762390+0000","last_became_active":"2023-10-11T10:43:41.767182+0000","last_became_peered":"2023-10-11T10:43:41.767182+0000","last_unstale":"2023-10-11T10:43:42.762390+0000","last_undegraded":"2023-10-11T10:43:42.762390+0000","last_fullsized":"2023-10-11T10:43:42.762390+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T13:33:09.638474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-10-11T10:43:43.804208+0000","last_change":"2023-10-11T10:43:41.772745+0000","last_active":"2023-10-11T10:43:43.804208+0000","last_peered":"2023-10-11T10:43:43.804208+0000","last_clean":"2023-10-11T10:43:43.804208+0000","last_became_active":"2023-10-11T10:43:41.772430+0000","last_became_peered":"2023-10-11T10:43:41.772430+0000","last_unstale":"2023-10-11T10:43:43.804208+0000","last_undegraded":"2023-10-11T10:43:43.804208+0000","last_fullsized":"2023-10-11T10:43:43.804208+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T18:51:15.753466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":141,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-10-11T10:43:41.764255+0000","last_change":"2023-10-11T10:43:37.764157+0000","last_active":"2023-10-11T10:43:41.764255+0000","last_peered":"2023-10-11T10:43:41.764255+0000","last_clean":"2023-10-11T10:43:41.764255+0000","last_became_active":"2023-10-11T10:43:37.762984+0000","last_became_peered":"2023-10-11T10:43:37.762984+0000","last_unstale":"2023-10-11T10:43:41.764255+0000","last_undegraded":"2023-10-11T10:43:41.764255+0000","last_fullsized":"2023-10-11T10:43:41.764255+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:36.739637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:36.739637+0000","last_clean_scrub_stamp":"2023-10-11T10:43:36.739637+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T19:04:42.047608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-10-11T10:43:42.762332+0000","last_change":"2023-10-11T10:43:41.772615+0000","last_active":"2023-10-11T10:43:42.762332+0000","last_peered":"2023-10-11T10:43:42.762332+0000","last_clean":"2023-10-11T10:43:42.762332+0000","last_became_active":"2023-10-11T10:43:41.772290+0000","last_became_peered":"2023-10-11T10:43:41.772290+0000","last_unstale":"2023-10-11T10:43:42.762332+0000","last_undegraded":"2023-10-11T10:43:42.762332+0000","last_fullsized":"2023-10-11T10:43:42.762332+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:26:06.909963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-10-11T10:43:43.804198+0000","last_change":"2023-10-11T10:43:41.772781+0000","last_active":"2023-10-11T10:43:43.804198+0000","last_peered":"2023-10-11T10:43:43.804198+0000","last_clean":"2023-10-11T10:43:43.804198+0000","last_became_active":"2023-10-11T10:43:41.772443+0000","last_became_peered":"2023-10-11T10:43:41.772443+0000","last_unstale":"2023-10-11T10:43:43.804198+0000","last_undegraded":"2023-10-11T10:43:43.804198+0000","last_fullsized":"2023-10-11T10:43:43.804198+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T14:15:30.547284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-10-11T10:43:43.804276+0000","last_change":"2023-10-11T10:43:41.767526+0000","last_active":"2023-10-11T10:43:43.804276+0000","last_peered":"2023-10-11T10:43:43.804276+0000","last_clean":"2023-10-11T10:43:43.804276+0000","last_became_active":"2023-10-11T10:43:41.767215+0000","last_became_peered":"2023-10-11T10:43:41.767215+0000","last_unstale":"2023-10-11T10:43:43.804276+0000","last_undegraded":"2023-10-11T10:43:43.804276+0000","last_fullsized":"2023-10-11T10:43:43.804276+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T15:44:46.953920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-10-11T10:43:42.762350+0000","last_change":"2023-10-11T10:43:41.767601+0000","last_active":"2023-10-11T10:43:42.762350+0000","last_peered":"2023-10-11T10:43:42.762350+0000","last_clean":"2023-10-11T10:43:42.762350+0000","last_became_active":"2023-10-11T10:43:41.767268+0000","last_became_peered":"2023-10-11T10:43:41.767268+0000","last_unstale":"2023-10-11T10:43:42.762350+0000","last_undegraded":"2023-10-11T10:43:42.762350+0000","last_fullsized":"2023-10-11T10:43:42.762350+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T20:13:11.818897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738373,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27148,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344692,"statfs":{"total":96636764160,"available":96608964608,"internally_reserved":0,"allocated":131072,"data_stored":37198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.224}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27200000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738375,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27728,"kb_used_data":708,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344112,"statfs":{"total":96636764160,"available":96608370688,"internally_reserved":0,"allocated":724992,"data_stored":627566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23200000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27200000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738374,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27728,"kb_used_data":708,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344112,"statfs":{"total":96636764160,"available":96608370688,"internally_reserved":0,"allocated":724992,"data_stored":627566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.31900000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34300000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-11T10:43:46.756 INFO:tasks.ceph.ceph_manager.cluster2:clean! 2023-10-11T10:43:46.756 INFO:tasks.ceph:Waiting until ceph cluster cluster2 is healthy... 2023-10-11T10:43:46.756 INFO:tasks.ceph.ceph_manager.cluster2:wait_until_healthy 2023-10-11T10:43:46.756 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 health --format=json 2023-10-11T10:43:47.117 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T10:43:47.117 INFO:teuthology.orchestra.run.smithi111.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-10-11T10:43:47.130 INFO:tasks.ceph.ceph_manager.cluster2:wait_until_healthy done 2023-10-11T10:43:47.131 INFO:teuthology.run_tasks:Running task exec... 2023-10-11T10:43:47.140 INFO:teuthology.task.exec:Executing custom commands... 2023-10-11T10:43:47.140 INFO:teuthology.task.exec:Running commands on role cluster1.client.mirror.0 host ubuntu@smithi111.front.sepia.ceph.com 2023-10-11T10:43:47.141 DEBUG:teuthology.orchestra.run.smithi111:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror mon '"'"'profile rbd-mirror-peer'"'"' osd '"'"'profile rbd'"'"'' 2023-10-11T10:43:47.548 INFO:teuthology.orchestra.run.smithi111.stdout:[client.mirror] 2023-10-11T10:43:47.548 INFO:teuthology.orchestra.run.smithi111.stdout: key = AQAafCZlDiQWAhAAw8ISseoz3kZ2nXMhyW3FOA== 2023-10-11T10:43:47.548 INFO:teuthology.orchestra.run.smithi111.stdout: caps mon = "profile rbd-mirror-peer" 2023-10-11T10:43:47.549 INFO:teuthology.orchestra.run.smithi111.stdout: caps osd = "profile rbd" 2023-10-11T10:43:47.550 INFO:teuthology.orchestra.run.smithi111.stderr:updated caps for client.mirror 2023-10-11T10:43:47.562 DEBUG:teuthology.orchestra.run.smithi111:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.0 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2023-10-11T10:43:47.970 INFO:teuthology.orchestra.run.smithi111.stdout:[client.mirror.0] 2023-10-11T10:43:47.970 INFO:teuthology.orchestra.run.smithi111.stdout: key = AQAafCZllj+pBRAAVfVJ0RH73M5kzEGU+iZccw== 2023-10-11T10:43:47.971 INFO:teuthology.orchestra.run.smithi111.stdout: caps mon = "profile rbd-mirror" 2023-10-11T10:43:47.971 INFO:teuthology.orchestra.run.smithi111.stdout: caps osd = "profile rbd" 2023-10-11T10:43:47.973 INFO:teuthology.orchestra.run.smithi111.stderr:updated caps for client.mirror.0 2023-10-11T10:43:47.985 DEBUG:teuthology.orchestra.run.smithi111:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.1 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2023-10-11T10:43:48.391 INFO:teuthology.orchestra.run.smithi111.stdout:[client.mirror.1] 2023-10-11T10:43:48.392 INFO:teuthology.orchestra.run.smithi111.stdout: key = AQAafCZl9/0DCRAA7gTZ6+bifx6zF/lYSzic4Q== 2023-10-11T10:43:48.392 INFO:teuthology.orchestra.run.smithi111.stdout: caps mon = "profile rbd-mirror" 2023-10-11T10:43:48.392 INFO:teuthology.orchestra.run.smithi111.stdout: caps osd = "profile rbd" 2023-10-11T10:43:48.394 INFO:teuthology.orchestra.run.smithi111.stderr:updated caps for client.mirror.1 2023-10-11T10:43:48.405 DEBUG:teuthology.orchestra.run.smithi111:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.2 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2023-10-11T10:43:48.813 INFO:teuthology.orchestra.run.smithi111.stdout:[client.mirror.2] 2023-10-11T10:43:48.813 INFO:teuthology.orchestra.run.smithi111.stdout: key = AQAafCZlxMOzDhAAkRbydTLt94oJ8u1ymnwzHg== 2023-10-11T10:43:48.814 INFO:teuthology.orchestra.run.smithi111.stdout: caps mon = "profile rbd-mirror" 2023-10-11T10:43:48.814 INFO:teuthology.orchestra.run.smithi111.stdout: caps osd = "profile rbd" 2023-10-11T10:43:48.816 INFO:teuthology.orchestra.run.smithi111.stderr:updated caps for client.mirror.2 2023-10-11T10:43:48.827 DEBUG:teuthology.orchestra.run.smithi111:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.3 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2023-10-11T10:43:49.234 INFO:teuthology.orchestra.run.smithi111.stdout:[client.mirror.3] 2023-10-11T10:43:49.235 INFO:teuthology.orchestra.run.smithi111.stdout: key = AQAafCZlG+ljFBAASq0jElIuWWFBidZPhhmgow== 2023-10-11T10:43:49.235 INFO:teuthology.orchestra.run.smithi111.stdout: caps mon = "profile rbd-mirror" 2023-10-11T10:43:49.235 INFO:teuthology.orchestra.run.smithi111.stdout: caps osd = "profile rbd" 2023-10-11T10:43:49.237 INFO:teuthology.orchestra.run.smithi111.stderr:updated caps for client.mirror.3 2023-10-11T10:43:49.249 DEBUG:teuthology.orchestra.run.smithi111:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror mon '"'"'profile rbd-mirror-peer'"'"' osd '"'"'profile rbd'"'"'' 2023-10-11T10:43:49.654 INFO:teuthology.orchestra.run.smithi111.stdout:[client.mirror] 2023-10-11T10:43:49.654 INFO:teuthology.orchestra.run.smithi111.stdout: key = AQA/fCZlh3vSARAAikuq2YV3q8iwr5PlIm8NtQ== 2023-10-11T10:43:49.654 INFO:teuthology.orchestra.run.smithi111.stdout: caps mon = "profile rbd-mirror-peer" 2023-10-11T10:43:49.655 INFO:teuthology.orchestra.run.smithi111.stdout: caps osd = "profile rbd" 2023-10-11T10:43:49.657 INFO:teuthology.orchestra.run.smithi111.stderr:updated caps for client.mirror 2023-10-11T10:43:49.669 DEBUG:teuthology.orchestra.run.smithi111:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.0 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2023-10-11T10:43:50.077 INFO:teuthology.orchestra.run.smithi111.stdout:[client.mirror.0] 2023-10-11T10:43:50.078 INFO:teuthology.orchestra.run.smithi111.stdout: key = AQA/fCZlrLKFBxAAtIcmebwDjJxnvs+6zwY+nA== 2023-10-11T10:43:50.078 INFO:teuthology.orchestra.run.smithi111.stdout: caps mon = "profile rbd-mirror" 2023-10-11T10:43:50.078 INFO:teuthology.orchestra.run.smithi111.stdout: caps osd = "profile rbd" 2023-10-11T10:43:50.080 INFO:teuthology.orchestra.run.smithi111.stderr:updated caps for client.mirror.0 2023-10-11T10:43:50.092 DEBUG:teuthology.orchestra.run.smithi111:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.1 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2023-10-11T10:43:50.499 INFO:teuthology.orchestra.run.smithi111.stdout:[client.mirror.1] 2023-10-11T10:43:50.499 INFO:teuthology.orchestra.run.smithi111.stdout: key = AQA/fCZl9F87DRAAxwv9S5pnPwjuVLpzMmjyeA== 2023-10-11T10:43:50.499 INFO:teuthology.orchestra.run.smithi111.stdout: caps mon = "profile rbd-mirror" 2023-10-11T10:43:50.499 INFO:teuthology.orchestra.run.smithi111.stdout: caps osd = "profile rbd" 2023-10-11T10:43:50.502 INFO:teuthology.orchestra.run.smithi111.stderr:updated caps for client.mirror.1 2023-10-11T10:43:50.513 DEBUG:teuthology.orchestra.run.smithi111:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.2 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2023-10-11T10:43:50.922 INFO:teuthology.orchestra.run.smithi111.stdout:[client.mirror.2] 2023-10-11T10:43:50.922 INFO:teuthology.orchestra.run.smithi111.stdout: key = AQA/fCZlP7UNExAAR7EYAsG+WkpNVC51Pve+zg== 2023-10-11T10:43:50.922 INFO:teuthology.orchestra.run.smithi111.stdout: caps mon = "profile rbd-mirror" 2023-10-11T10:43:50.922 INFO:teuthology.orchestra.run.smithi111.stdout: caps osd = "profile rbd" 2023-10-11T10:43:50.925 INFO:teuthology.orchestra.run.smithi111.stderr:updated caps for client.mirror.2 2023-10-11T10:43:50.936 DEBUG:teuthology.orchestra.run.smithi111:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.3 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2023-10-11T10:43:51.343 INFO:teuthology.orchestra.run.smithi111.stdout:[client.mirror.3] 2023-10-11T10:43:51.344 INFO:teuthology.orchestra.run.smithi111.stdout: key = AQA/fCZlRbRVFhAABrKtvCT9iJoU7LA1u3kEsA== 2023-10-11T10:43:51.344 INFO:teuthology.orchestra.run.smithi111.stdout: caps mon = "profile rbd-mirror" 2023-10-11T10:43:51.344 INFO:teuthology.orchestra.run.smithi111.stdout: caps osd = "profile rbd" 2023-10-11T10:43:51.346 INFO:teuthology.orchestra.run.smithi111.stderr:updated caps for client.mirror.3 2023-10-11T10:43:51.358 INFO:teuthology.run_tasks:Running task rbd-mirror... 2023-10-11T10:43:51.369 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2023-10-11T10:43:51.370 DEBUG:teuthology.orchestra.run.smithi111:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2023-10-11T10:43:51.372 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2023-10-11T10:43:51.373 INFO:teuthology.run_tasks:Running task rbd-mirror... 2023-10-11T10:43:51.381 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2023-10-11T10:43:51.381 DEBUG:teuthology.orchestra.run.smithi111:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2023-10-11T10:43:51.383 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2023-10-11T10:43:51.383 INFO:teuthology.run_tasks:Running task rbd-mirror... 2023-10-11T10:43:51.391 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Restarting daemon 2023-10-11T10:43:51.391 DEBUG:teuthology.orchestra.run.smithi111:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 2023-10-11T10:43:51.394 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Started 2023-10-11T10:43:51.394 INFO:teuthology.run_tasks:Running task rbd-mirror... 2023-10-11T10:43:51.402 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Restarting daemon 2023-10-11T10:43:51.402 DEBUG:teuthology.orchestra.run.smithi111:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 2023-10-11T10:43:51.405 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Started 2023-10-11T10:43:51.405 INFO:teuthology.run_tasks:Running task rbd-mirror... 2023-10-11T10:43:51.413 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2023-10-11T10:43:51.414 DEBUG:teuthology.orchestra.run.smithi111:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2023-10-11T10:43:51.416 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2023-10-11T10:43:51.416 INFO:teuthology.run_tasks:Running task rbd-mirror... 2023-10-11T10:43:51.424 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2023-10-11T10:43:51.424 DEBUG:teuthology.orchestra.run.smithi111:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2023-10-11T10:43:51.427 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2023-10-11T10:43:51.427 INFO:teuthology.run_tasks:Running task rbd-mirror... 2023-10-11T10:43:51.435 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Restarting daemon 2023-10-11T10:43:51.436 DEBUG:teuthology.orchestra.run.smithi111:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 2023-10-11T10:43:51.442 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Started 2023-10-11T10:43:51.443 INFO:teuthology.run_tasks:Running task rbd-mirror... 2023-10-11T10:43:51.451 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Restarting daemon 2023-10-11T10:43:51.451 DEBUG:teuthology.orchestra.run.smithi111:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 2023-10-11T10:43:51.457 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Started 2023-10-11T10:43:51.458 INFO:teuthology.run_tasks:Running task rbd-mirror-thrash... 2023-10-11T10:43:51.469 INFO:tasks.rbd_mirror_thrash:rbd_mirror_thrash using random seed: 1697021031 2023-10-11T10:43:51.470 DEBUG:tasks.rbd_mirror_thrash:Yielding 2023-10-11T10:43:51.470 INFO:teuthology.run_tasks:Running task rbd-mirror-thrash... 2023-10-11T10:43:51.478 INFO:tasks.rbd_mirror_thrash:rbd_mirror_thrash using random seed: 1697021031 2023-10-11T10:43:51.479 DEBUG:tasks.rbd_mirror_thrash:Yielding 2023-10-11T10:43:51.479 INFO:teuthology.run_tasks:Running task workunit... 2023-10-11T10:43:51.490 INFO:tasks.workunit:Pulling workunits from ref 3d099a3859cc211cf8dd2acdaf3194ba0e96ea9c 2023-10-11T10:43:51.491 INFO:tasks.workunit:Making a separate scratch dir for every client... 2023-10-11T10:43:51.491 DEBUG:teuthology.orchestra.run.smithi111:> stat -- /home/ubuntu/cephtest/mnt.cluster1.mirror 2023-10-11T10:43:51.491 INFO:tasks.rbd_mirror_thrash:starting thrash for cluster cluster1 2023-10-11T10:43:51.492 INFO:tasks.rbd_mirror_thrash:waiting for 107 secs before thrashing 2023-10-11T10:43:51.492 INFO:tasks.rbd_mirror_thrash:starting thrash for cluster cluster2 2023-10-11T10:43:51.492 INFO:tasks.rbd_mirror_thrash:waiting for 115 secs before thrashing 2023-10-11T10:43:51.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-11T10:43:51.530 INFO:teuthology.orchestra.run.smithi111.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.cluster1.mirror': No such file or directory 2023-10-11T10:43:51.530 DEBUG:teuthology.orchestra.run.smithi111:> mkdir -- /home/ubuntu/cephtest/mnt.cluster1.mirror 2023-10-11T10:43:51.586 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.cluster1.mirror 2023-10-11T10:43:51.586 DEBUG:teuthology.orchestra.run.smithi111:> cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror && mkdir -- client.mirror 2023-10-11T10:43:51.643 INFO:tasks.workunit:timeout=6h 2023-10-11T10:43:51.643 INFO:tasks.workunit:cleanup=True 2023-10-11T10:43:51.643 DEBUG:teuthology.orchestra.run.smithi111:> rm -rf /home/ubuntu/cephtest/clone.cluster1.client.mirror && git clone https://github.com/idryomov/ceph /home/ubuntu/cephtest/clone.cluster1.client.mirror && cd /home/ubuntu/cephtest/clone.cluster1.client.mirror && git checkout 3d099a3859cc211cf8dd2acdaf3194ba0e96ea9c 2023-10-11T10:43:51.701 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:Cloning into '/home/ubuntu/cephtest/clone.cluster1.client.mirror'... 2023-10-11T10:44:27.969 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:Note: switching to '3d099a3859cc211cf8dd2acdaf3194ba0e96ea9c'. 2023-10-11T10:44:27.969 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: 2023-10-11T10:44:27.970 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2023-10-11T10:44:27.970 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:changes and commit them, and you can discard any commits you make in this 2023-10-11T10:44:27.970 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:state without impacting any branches by switching back to a branch. 2023-10-11T10:44:27.970 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: 2023-10-11T10:44:27.971 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:If you want to create a new branch to retain commits you create, you may 2023-10-11T10:44:27.971 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:do so (now or later) by using -c with the switch command. Example: 2023-10-11T10:44:27.971 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: 2023-10-11T10:44:27.971 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: git switch -c 2023-10-11T10:44:27.972 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: 2023-10-11T10:44:27.972 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:Or undo this operation with: 2023-10-11T10:44:27.972 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: 2023-10-11T10:44:27.972 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: git switch - 2023-10-11T10:44:27.972 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: 2023-10-11T10:44:27.973 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2023-10-11T10:44:27.973 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: 2023-10-11T10:44:27.973 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:HEAD is now at 3d099a3859c Merge branch 'wip-63149' of https://github.com/idryomov/ceph into HEAD 2023-10-11T10:44:27.975 DEBUG:teuthology.orchestra.run.smithi111:> cd -- /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.cluster1.client.mirror 2023-10-11T10:44:28.043 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2023-10-11T10:44:28.045 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/direct_io' 2023-10-11T10:44:28.046 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2023-10-11T10:44:28.604 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2023-10-11T10:44:28.664 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2023-10-11T10:44:28.702 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/direct_io' 2023-10-11T10:44:28.704 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/fs' 2023-10-11T10:44:28.704 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2023-10-11T10:44:28.740 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/fs' 2023-10-11T10:44:28.744 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2023-10-11T10:44:28.744 DEBUG:teuthology.orchestra.run.smithi111:> dd if=/home/ubuntu/cephtest/workunits.list.cluster1.client.mirror of=/dev/stdout 2023-10-11T10:44:28.800 INFO:tasks.workunit:Running workunits matching rbd/rbd_mirror_stress.sh on cluster1.client.mirror... 2023-10-11T10:44:28.802 INFO:tasks.workunit:Running workunit rbd/rbd_mirror_stress.sh... 2023-10-11T10:44:28.802 DEBUG:teuthology.orchestra.run.smithi111:workunit test rbd/rbd_mirror_stress.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3d099a3859cc211cf8dd2acdaf3194ba0e96ea9c TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS='' MIRROR_IMAGE_MODE=snapshot MIRROR_POOL_MODE=image RBD_IMAGE_FEATURES=layering,exclusive-lock,object-map,fast-diff RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_stress.sh 2023-10-11T10:44:28.861 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ IMAGE_COUNT=50 2023-10-11T10:44:28.861 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ export LOCKDEP=0 2023-10-11T10:44:28.862 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ LOCKDEP=0 2023-10-11T10:44:28.862 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ dirname /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_stress.sh 2023-10-11T10:44:28.864 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ . /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_helpers.sh 2023-10-11T10:44:28.864 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ type xmlstarlet 2023-10-11T10:44:28.865 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ XMLSTARLET=xmlstarlet 2023-10-11T10:44:28.865 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ RBD_MIRROR_INSTANCES=4 2023-10-11T10:44:28.865 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ CLUSTER1=cluster1 2023-10-11T10:44:28.865 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ CLUSTER2=cluster2 2023-10-11T10:44:28.866 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ PEER_CLUSTER_SUFFIX= 2023-10-11T10:44:28.866 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ POOL=mirror 2023-10-11T10:44:28.866 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ PARENT_POOL=mirror_parent 2023-10-11T10:44:28.866 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ NS1=ns1 2023-10-11T10:44:28.867 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ NS2=ns2 2023-10-11T10:44:28.867 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ TEMPDIR= 2023-10-11T10:44:28.867 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ CEPH_ID=mirror 2023-10-11T10:44:28.867 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ RBD_IMAGE_FEATURES=layering,exclusive-lock,object-map,fast-diff 2023-10-11T10:44:28.867 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ MIRROR_USER_ID_PREFIX=mirror. 2023-10-11T10:44:28.868 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ MIRROR_POOL_MODE=image 2023-10-11T10:44:28.868 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ MIRROR_IMAGE_MODE=snapshot 2023-10-11T10:44:28.868 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ export 'CEPH_ARGS=--id mirror' 2023-10-11T10:44:28.868 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ CEPH_ARGS='--id mirror' 2023-10-11T10:44:28.869 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ LAST_MIRROR_INSTANCE=3 2023-10-11T10:44:28.869 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ dirname /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_stress.sh 2023-10-11T10:44:28.869 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ readlink -f /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/../../../src 2023-10-11T10:44:28.869 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror/src 2023-10-11T10:44:28.869 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ CEPH_BIN=. 2023-10-11T10:44:28.870 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ CEPH_SRC=. 2023-10-11T10:44:28.870 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' -e CMakeCache.txt ']' 2023-10-11T10:44:28.872 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' 0 -gt 0 ']' 2023-10-11T10:44:28.872 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ setup 2023-10-11T10:44:28.872 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local c 2023-10-11T10:44:28.872 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ trap 'cleanup $?' INT TERM EXIT 2023-10-11T10:44:28.873 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ setup_tempdir 2023-10-11T10:44:28.873 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n '' ']' 2023-10-11T10:44:28.873 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ mktemp -d 2023-10-11T10:44:28.873 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ TEMPDIR=/tmp/tmp.nGgfBEp0YO 2023-10-11T10:44:28.874 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -z 1 ']' 2023-10-11T10:44:28.874 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ setup_pools cluster1 cluster2 2023-10-11T10:44:28.874 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T10:44:28.874 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local remote_cluster=cluster2 2023-10-11T10:44:28.875 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local mon_map_file 2023-10-11T10:44:28.875 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local mon_addr 2023-10-11T10:44:28.875 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local admin_key_file 2023-10-11T10:44:28.875 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local uuid 2023-10-11T10:44:28.876 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ CEPH_ARGS= 2023-10-11T10:44:28.876 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ ceph --cluster cluster1 osd pool create mirror 64 64 2023-10-11T10:44:29.701 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:pool 'mirror' already exists 2023-10-11T10:44:29.710 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ CEPH_ARGS= 2023-10-11T10:44:29.710 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ ceph --cluster cluster1 osd pool create mirror_parent 64 64 2023-10-11T10:44:30.703 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:pool 'mirror_parent' already exists 2023-10-11T10:44:30.713 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ CEPH_ARGS= 2023-10-11T10:44:30.713 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 pool init mirror 2023-10-11T10:44:33.604 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ CEPH_ARGS= 2023-10-11T10:44:33.604 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 pool init mirror_parent 2023-10-11T10:44:36.616 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n '' ']' 2023-10-11T10:44:36.616 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ CEPH_ARGS= 2023-10-11T10:44:36.617 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 mirror pool enable --site-name cluster1 mirror image 2023-10-11T10:44:36.661 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 mirror pool enable mirror_parent image 2023-10-11T10:44:36.702 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 namespace create mirror/ns1 2023-10-11T10:44:36.738 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 namespace create mirror/ns2 2023-10-11T10:44:36.773 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 mirror pool enable mirror/ns1 image 2023-10-11T10:44:36.812 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 mirror pool enable mirror/ns2 image 2023-10-11T10:44:36.851 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -z ']' 2023-10-11T10:44:36.851 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -z ']' 2023-10-11T10:44:36.851 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ peer_add cluster1 mirror cluster2 2023-10-11T10:44:36.852 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T10:44:36.852 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:44:36.852 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:44:36.852 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:44:36.852 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local client_cluster=cluster2 2023-10-11T10:44:36.852 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:44:36.853 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local remote_cluster=cluster2 2023-10-11T10:44:36.853 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local uuid_var_name 2023-10-11T10:44:36.853 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n '' ']' 2023-10-11T10:44:36.853 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local error_code 2023-10-11T10:44:36.853 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local peer_uuid 2023-10-11T10:44:36.854 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 1 2 4 8 16 32 2023-10-11T10:44:36.854 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ set +e 2023-10-11T10:44:36.854 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ rbd --cluster cluster1 mirror pool peer add mirror cluster2 2023-10-11T10:44:36.888 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ peer_uuid=44545119-3275-4365-8b55-eb5090e6b339 2023-10-11T10:44:36.889 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ error_code=0 2023-10-11T10:44:36.889 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ set -e 2023-10-11T10:44:36.889 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' 0 -eq 17 ']' 2023-10-11T10:44:36.889 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 0 -eq 0 2023-10-11T10:44:36.889 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n '' ']' 2023-10-11T10:44:36.890 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T10:44:36.890 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ peer_add cluster1 mirror_parent cluster2 2023-10-11T10:44:36.890 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T10:44:36.890 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:44:36.890 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror_parent 2023-10-11T10:44:36.890 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:44:36.891 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local client_cluster=cluster2 2023-10-11T10:44:36.891 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:44:36.891 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local remote_cluster=cluster2 2023-10-11T10:44:36.891 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local uuid_var_name 2023-10-11T10:44:36.891 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n '' ']' 2023-10-11T10:44:36.891 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local error_code 2023-10-11T10:44:36.892 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local peer_uuid 2023-10-11T10:44:36.892 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 1 2 4 8 16 32 2023-10-11T10:44:36.892 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ set +e 2023-10-11T10:44:36.892 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ rbd --cluster cluster1 mirror pool peer add mirror_parent cluster2 2023-10-11T10:44:36.925 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ peer_uuid=7e5d1488-45f2-4507-a6eb-8d60fb3bcb97 2023-10-11T10:44:36.925 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ error_code=0 2023-10-11T10:44:36.925 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ set -e 2023-10-11T10:44:36.926 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' 0 -eq 17 ']' 2023-10-11T10:44:36.926 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 0 -eq 0 2023-10-11T10:44:36.926 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n '' ']' 2023-10-11T10:44:36.926 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T10:44:36.926 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ setup_pools cluster2 cluster1 2023-10-11T10:44:36.926 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:44:36.927 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local remote_cluster=cluster1 2023-10-11T10:44:36.927 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local mon_map_file 2023-10-11T10:44:36.927 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local mon_addr 2023-10-11T10:44:36.927 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local admin_key_file 2023-10-11T10:44:36.927 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local uuid 2023-10-11T10:44:36.928 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ CEPH_ARGS= 2023-10-11T10:44:36.928 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ ceph --cluster cluster2 osd pool create mirror 64 64 2023-10-11T10:44:37.915 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:pool 'mirror' already exists 2023-10-11T10:44:37.924 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ CEPH_ARGS= 2023-10-11T10:44:37.925 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ ceph --cluster cluster2 osd pool create mirror_parent 64 64 2023-10-11T10:44:38.936 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:pool 'mirror_parent' already exists 2023-10-11T10:44:38.946 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ CEPH_ARGS= 2023-10-11T10:44:38.946 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 pool init mirror 2023-10-11T10:44:41.812 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ CEPH_ARGS= 2023-10-11T10:44:41.813 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 pool init mirror_parent 2023-10-11T10:44:44.824 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n '' ']' 2023-10-11T10:44:44.825 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ CEPH_ARGS= 2023-10-11T10:44:44.825 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 mirror pool enable --site-name cluster2 mirror image 2023-10-11T10:44:44.875 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 mirror pool enable mirror_parent image 2023-10-11T10:44:44.913 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 namespace create mirror/ns1 2023-10-11T10:44:44.948 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 namespace create mirror/ns2 2023-10-11T10:44:44.983 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 mirror pool enable mirror/ns1 image 2023-10-11T10:44:45.019 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 mirror pool enable mirror/ns2 image 2023-10-11T10:44:45.059 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -z ']' 2023-10-11T10:44:45.059 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -z ']' 2023-10-11T10:44:45.059 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ peer_add cluster2 mirror cluster1 2023-10-11T10:44:45.059 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:44:45.059 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:44:45.060 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:44:45.060 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:44:45.060 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local client_cluster=cluster1 2023-10-11T10:44:45.060 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:44:45.060 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local remote_cluster=cluster1 2023-10-11T10:44:45.061 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local uuid_var_name 2023-10-11T10:44:45.061 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n '' ']' 2023-10-11T10:44:45.061 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local error_code 2023-10-11T10:44:45.061 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local peer_uuid 2023-10-11T10:44:45.061 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 1 2 4 8 16 32 2023-10-11T10:44:45.061 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ set +e 2023-10-11T10:44:45.062 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ rbd --cluster cluster2 mirror pool peer add mirror cluster1 2023-10-11T10:44:45.095 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ peer_uuid=fac57d8f-1ab6-4c92-ab7d-bc0209748be4 2023-10-11T10:44:45.095 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ error_code=0 2023-10-11T10:44:45.095 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ set -e 2023-10-11T10:44:45.095 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' 0 -eq 17 ']' 2023-10-11T10:44:45.095 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 0 -eq 0 2023-10-11T10:44:45.096 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n '' ']' 2023-10-11T10:44:45.096 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T10:44:45.096 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ peer_add cluster2 mirror_parent cluster1 2023-10-11T10:44:45.096 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:44:45.096 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:44:45.097 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror_parent 2023-10-11T10:44:45.097 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:44:45.097 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local client_cluster=cluster1 2023-10-11T10:44:45.097 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:44:45.097 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local remote_cluster=cluster1 2023-10-11T10:44:45.097 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local uuid_var_name 2023-10-11T10:44:45.098 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n '' ']' 2023-10-11T10:44:45.098 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local error_code 2023-10-11T10:44:45.098 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local peer_uuid 2023-10-11T10:44:45.098 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 1 2 4 8 16 32 2023-10-11T10:44:45.098 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ set +e 2023-10-11T10:44:45.098 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ rbd --cluster cluster2 mirror pool peer add mirror_parent cluster1 2023-10-11T10:44:45.130 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ peer_uuid=216c798c-be92-4e54-a14c-4f00e5085964 2023-10-11T10:44:45.130 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ error_code=0 2023-10-11T10:44:45.131 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ set -e 2023-10-11T10:44:45.131 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' 0 -eq 17 ']' 2023-10-11T10:44:45.131 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 0 -eq 0 2023-10-11T10:44:45.131 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n '' ']' 2023-10-11T10:44:45.131 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T10:44:45.132 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n '' ']' 2023-10-11T10:44:45.132 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ start_mirrors cluster1 2023-10-11T10:44:45.132 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T10:44:45.132 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ seq 0 3 2023-10-11T10:44:45.155 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-10-11T10:44:45.156 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ start_mirror cluster1:0 2023-10-11T10:44:45.156 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1:0 2023-10-11T10:44:45.156 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local instance 2023-10-11T10:44:45.156 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-10-11T10:44:45.157 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local val=cluster1:0 2023-10-11T10:44:45.157 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster_var_name=cluster 2023-10-11T10:44:45.157 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local instance_var_name=instance 2023-10-11T10:44:45.157 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ cluster=cluster1 2023-10-11T10:44:45.157 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ instance=0 2023-10-11T10:44:45.157 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' 0 = cluster1:0 ']' 2023-10-11T10:44:45.158 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ eval cluster=cluster1 2023-10-11T10:44:45.158 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ cluster=cluster1 2023-10-11T10:44:45.158 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ eval instance=0 2023-10-11T10:44:45.158 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ instance=0 2023-10-11T10:44:45.158 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test -n 1 2023-10-11T10:44:45.159 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 2023-10-11T10:44:45.159 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-10-11T10:44:45.159 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ start_mirror cluster1:1 2023-10-11T10:44:45.159 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1:1 2023-10-11T10:44:45.159 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local instance 2023-10-11T10:44:45.159 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-10-11T10:44:45.160 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local val=cluster1:1 2023-10-11T10:44:45.160 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster_var_name=cluster 2023-10-11T10:44:45.160 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local instance_var_name=instance 2023-10-11T10:44:45.160 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ cluster=cluster1 2023-10-11T10:44:45.160 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ instance=1 2023-10-11T10:44:45.160 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' 1 = cluster1:1 ']' 2023-10-11T10:44:45.161 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ eval cluster=cluster1 2023-10-11T10:44:45.161 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ cluster=cluster1 2023-10-11T10:44:45.161 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ eval instance=1 2023-10-11T10:44:45.161 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ instance=1 2023-10-11T10:44:45.161 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test -n 1 2023-10-11T10:44:45.162 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 2023-10-11T10:44:45.162 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-10-11T10:44:45.162 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ start_mirror cluster1:2 2023-10-11T10:44:45.162 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1:2 2023-10-11T10:44:45.162 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local instance 2023-10-11T10:44:45.162 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-10-11T10:44:45.163 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local val=cluster1:2 2023-10-11T10:44:45.163 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster_var_name=cluster 2023-10-11T10:44:45.163 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local instance_var_name=instance 2023-10-11T10:44:45.163 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ cluster=cluster1 2023-10-11T10:44:45.163 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ instance=2 2023-10-11T10:44:45.164 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' 2 = cluster1:2 ']' 2023-10-11T10:44:45.164 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ eval cluster=cluster1 2023-10-11T10:44:45.164 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ cluster=cluster1 2023-10-11T10:44:45.164 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ eval instance=2 2023-10-11T10:44:45.164 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ instance=2 2023-10-11T10:44:45.164 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test -n 1 2023-10-11T10:44:45.165 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 2023-10-11T10:44:45.165 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-10-11T10:44:45.165 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ start_mirror cluster1:3 2023-10-11T10:44:45.165 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1:3 2023-10-11T10:44:45.165 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local instance 2023-10-11T10:44:45.165 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ set_cluster_instance cluster1:3 cluster instance 2023-10-11T10:44:45.166 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local val=cluster1:3 2023-10-11T10:44:45.166 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster_var_name=cluster 2023-10-11T10:44:45.166 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local instance_var_name=instance 2023-10-11T10:44:45.166 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ cluster=cluster1 2023-10-11T10:44:45.166 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ instance=3 2023-10-11T10:44:45.167 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' 3 = cluster1:3 ']' 2023-10-11T10:44:45.167 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ eval cluster=cluster1 2023-10-11T10:44:45.167 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ cluster=cluster1 2023-10-11T10:44:45.167 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ eval instance=3 2023-10-11T10:44:45.167 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ instance=3 2023-10-11T10:44:45.167 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test -n 1 2023-10-11T10:44:45.168 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 2023-10-11T10:44:45.168 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ start_mirrors cluster2 2023-10-11T10:44:45.168 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:44:45.168 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ seq 0 3 2023-10-11T10:44:45.168 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-10-11T10:44:45.168 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ start_mirror cluster2:0 2023-10-11T10:44:45.169 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2:0 2023-10-11T10:44:45.169 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local instance 2023-10-11T10:44:45.169 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ set_cluster_instance cluster2:0 cluster instance 2023-10-11T10:44:45.169 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local val=cluster2:0 2023-10-11T10:44:45.169 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster_var_name=cluster 2023-10-11T10:44:45.169 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local instance_var_name=instance 2023-10-11T10:44:45.170 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ cluster=cluster2 2023-10-11T10:44:45.170 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ instance=0 2023-10-11T10:44:45.170 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' 0 = cluster2:0 ']' 2023-10-11T10:44:45.170 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ eval cluster=cluster2 2023-10-11T10:44:45.170 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ cluster=cluster2 2023-10-11T10:44:45.170 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ eval instance=0 2023-10-11T10:44:45.171 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ instance=0 2023-10-11T10:44:45.171 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test -n 1 2023-10-11T10:44:45.171 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 2023-10-11T10:44:45.171 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-10-11T10:44:45.171 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ start_mirror cluster2:1 2023-10-11T10:44:45.171 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2:1 2023-10-11T10:44:45.172 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local instance 2023-10-11T10:44:45.172 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ set_cluster_instance cluster2:1 cluster instance 2023-10-11T10:44:45.172 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local val=cluster2:1 2023-10-11T10:44:45.172 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster_var_name=cluster 2023-10-11T10:44:45.172 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local instance_var_name=instance 2023-10-11T10:44:45.173 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ cluster=cluster2 2023-10-11T10:44:45.173 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ instance=1 2023-10-11T10:44:45.173 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' 1 = cluster2:1 ']' 2023-10-11T10:44:45.173 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ eval cluster=cluster2 2023-10-11T10:44:45.173 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ cluster=cluster2 2023-10-11T10:44:45.173 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ eval instance=1 2023-10-11T10:44:45.174 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ instance=1 2023-10-11T10:44:45.174 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test -n 1 2023-10-11T10:44:45.174 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 2023-10-11T10:44:45.174 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-10-11T10:44:45.174 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ start_mirror cluster2:2 2023-10-11T10:44:45.175 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2:2 2023-10-11T10:44:45.175 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local instance 2023-10-11T10:44:45.175 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ set_cluster_instance cluster2:2 cluster instance 2023-10-11T10:44:45.175 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local val=cluster2:2 2023-10-11T10:44:45.175 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster_var_name=cluster 2023-10-11T10:44:45.175 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local instance_var_name=instance 2023-10-11T10:44:45.176 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ cluster=cluster2 2023-10-11T10:44:45.176 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ instance=2 2023-10-11T10:44:45.176 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' 2 = cluster2:2 ']' 2023-10-11T10:44:45.176 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ eval cluster=cluster2 2023-10-11T10:44:45.176 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ cluster=cluster2 2023-10-11T10:44:45.177 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ eval instance=2 2023-10-11T10:44:45.177 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ instance=2 2023-10-11T10:44:45.177 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test -n 1 2023-10-11T10:44:45.177 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 2023-10-11T10:44:45.177 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-10-11T10:44:45.177 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ start_mirror cluster2:3 2023-10-11T10:44:45.178 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2:3 2023-10-11T10:44:45.178 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local instance 2023-10-11T10:44:45.178 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ set_cluster_instance cluster2:3 cluster instance 2023-10-11T10:44:45.178 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local val=cluster2:3 2023-10-11T10:44:45.178 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster_var_name=cluster 2023-10-11T10:44:45.179 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local instance_var_name=instance 2023-10-11T10:44:45.179 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ cluster=cluster2 2023-10-11T10:44:45.179 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ instance=3 2023-10-11T10:44:45.179 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' 3 = cluster2:3 ']' 2023-10-11T10:44:45.179 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ eval cluster=cluster2 2023-10-11T10:44:45.179 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ cluster=cluster2 2023-10-11T10:44:45.180 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ eval instance=3 2023-10-11T10:44:45.180 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ instance=3 2023-10-11T10:44:45.180 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test -n 1 2023-10-11T10:44:45.180 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 2023-10-11T10:44:45.180 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ testlog 'TEST: add image and test replay after client crashes' 2023-10-11T10:44:45.180 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ tee -a /tmp/tmp.nGgfBEp0YO/rbd-mirror.test.log 2023-10-11T10:44:45.180 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ date '+%F %T' 2023-10-11T10:44:45.181 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ echo 2023-10-11 10:44:45 TEST: add image and test replay after client crashes 2023-10-11T10:44:45.181 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:2023-10-11 10:44:45 TEST: add image and test replay after client crashes 2023-10-11T10:44:45.181 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=test 2023-10-11T10:44:45.181 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ create_image_and_enable_mirror cluster2 mirror test snapshot 512M 2023-10-11T10:44:45.181 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:44:45.181 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:44:45.182 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:44:45.182 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:44:45.182 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:44:45.182 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:44:45.182 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local mode=snapshot 2023-10-11T10:44:45.183 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n snapshot ']' 2023-10-11T10:44:45.183 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:44:45.183 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ create_image cluster2 mirror test 512M 2023-10-11T10:44:45.183 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:44:45.183 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:44:45.183 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:44:45.184 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:44:45.184 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:44:45.184 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:44:45.184 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local size=128 2023-10-11T10:44:45.184 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n 512M ']' 2023-10-11T10:44:45.184 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ size=512M 2023-10-11T10:44:45.185 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:44:45.185 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/test 2023-10-11T10:44:45.195 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' image = image ']' 2023-10-11T10:44:45.195 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ enable_mirror cluster2 mirror test snapshot 2023-10-11T10:44:45.195 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:44:45.195 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:44:45.196 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:44:45.196 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local mode=snapshot 2023-10-11T10:44:45.196 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/test snapshot 2023-10-11T10:44:45.882 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:Mirroring enabled 2023-10-11T10:44:45.889 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 info mirror/test 2023-10-11T10:44:45.925 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:rbd image 'test': 2023-10-11T10:44:45.925 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: size 512 MiB in 128 objects 2023-10-11T10:44:45.925 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: order 22 (4 MiB objects) 2023-10-11T10:44:45.925 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: snapshot_count: 1 2023-10-11T10:44:45.925 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: id: 10a4afc22214 2023-10-11T10:44:45.925 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: block_name_prefix: rbd_data.10a4afc22214 2023-10-11T10:44:45.926 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: format: 2 2023-10-11T10:44:45.926 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-10-11T10:44:45.926 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: op_features: 2023-10-11T10:44:45.926 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: flags: 2023-10-11T10:44:45.926 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: create_timestamp: Wed Oct 11 10:44:45 2023 2023-10-11T10:44:45.927 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: access_timestamp: Wed Oct 11 10:44:45 2023 2023-10-11T10:44:45.927 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: modify_timestamp: Wed Oct 11 10:44:45 2023 2023-10-11T10:44:45.927 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring state: enabled 2023-10-11T10:44:45.927 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring mode: snapshot 2023-10-11T10:44:45.927 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring global id: 3dd93db4-c9cc-462c-8b49-007eba9a90d2 2023-10-11T10:44:45.927 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring primary: true 2023-10-11T10:44:45.928 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_replay_started cluster1 mirror test 2023-10-11T10:44:45.929 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T10:44:45.929 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:44:45.929 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:44:45.929 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2023-10-11T10:44:45.929 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T10:44:45.929 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:44:45.930 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:44:45.930 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local state=started 2023-10-11T10:44:45.930 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T10:44:45.930 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-10-11T10:44:45.930 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 1 2023-10-11T10:44:46.930 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test_image_replay_state cluster1 mirror test started 2023-10-11T10:44:46.930 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T10:44:46.930 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:44:46.931 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:44:46.931 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local test_state=started 2023-10-11T10:44:46.931 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local status_result 2023-10-11T10:44:46.931 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_state=stopped 2023-10-11T10:44:46.931 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2023-10-11T10:44:46.932 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_instance=cluster1 2023-10-11T10:44:46.932 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T10:44:46.932 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T10:44:46.932 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ grep -i state 2023-10-11T10:44:46.933 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance=cluster1 2023-10-11T10:44:46.933 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local loop_instance 2023-10-11T10:44:46.933 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-10-11T10:44:46.933 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sleep 0 2023-10-11T10:44:46.936 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-10-11T10:44:46.936 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ seq 0 3 2023-10-11T10:44:46.936 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-10-11T10:44:46.937 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-10-11T10:44:46.937 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1:0 2023-10-11T10:44:46.937 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T10:44:46.937 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance 2023-10-11T10:44:46.937 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-10-11T10:44:46.938 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local val=cluster1:0 2023-10-11T10:44:46.938 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_var_name=cluster 2023-10-11T10:44:46.938 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance_var_name=instance 2023-10-11T10:44:46.938 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ cluster=cluster1 2023-10-11T10:44:46.938 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ instance=0 2023-10-11T10:44:46.939 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' 0 = cluster1:0 ']' 2023-10-11T10:44:46.939 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval cluster=cluster1 2023-10-11T10:44:46.939 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T10:44:46.939 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval instance=0 2023-10-11T10:44:46.939 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=0 2023-10-11T10:44:46.940 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-10-11T10:44:46.940 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local local_cluster=cluster1:0 2023-10-11T10:44:46.940 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster=cluster1 2023-10-11T10:44:46.940 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance 2023-10-11T10:44:46.940 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-10-11T10:44:46.940 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local val=cluster1:0 2023-10-11T10:44:46.941 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster_var_name=local_cluster 2023-10-11T10:44:46.941 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance_var_name=instance 2023-10-11T10:44:46.941 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T10:44:46.941 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=0 2023-10-11T10:44:46.941 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ '[' 0 = cluster1:0 ']' 2023-10-11T10:44:46.942 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval local_cluster=cluster1 2023-10-11T10:44:46.942 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ local_cluster=cluster1 2023-10-11T10:44:46.942 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval instance=0 2023-10-11T10:44:46.942 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ instance=0 2023-10-11T10:44:46.942 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-10-11T10:44:46.956 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T10:44:46.956 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T10:44:46.956 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T10:44:46.956 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-10-11T10:44:47.056 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-10-11T10:44:47.057 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd mirror status : get status for rbd mirror 2023-10-11T10:44:47.057 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:admin_socket: invalid command 2023-10-11T10:44:47.060 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-10-11T10:44:47.060 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-10-11T10:44:47.060 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1:1 2023-10-11T10:44:47.061 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T10:44:47.061 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance 2023-10-11T10:44:47.061 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-10-11T10:44:47.061 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local val=cluster1:1 2023-10-11T10:44:47.062 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_var_name=cluster 2023-10-11T10:44:47.062 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance_var_name=instance 2023-10-11T10:44:47.062 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ cluster=cluster1 2023-10-11T10:44:47.062 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ instance=1 2023-10-11T10:44:47.062 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' 1 = cluster1:1 ']' 2023-10-11T10:44:47.063 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval cluster=cluster1 2023-10-11T10:44:47.063 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T10:44:47.063 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval instance=1 2023-10-11T10:44:47.063 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=1 2023-10-11T10:44:47.063 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-10-11T10:44:47.064 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local local_cluster=cluster1:1 2023-10-11T10:44:47.064 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster=cluster1 2023-10-11T10:44:47.064 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance 2023-10-11T10:44:47.064 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-10-11T10:44:47.064 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local val=cluster1:1 2023-10-11T10:44:47.065 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster_var_name=local_cluster 2023-10-11T10:44:47.065 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance_var_name=instance 2023-10-11T10:44:47.065 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T10:44:47.065 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=1 2023-10-11T10:44:47.065 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ '[' 1 = cluster1:1 ']' 2023-10-11T10:44:47.066 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval local_cluster=cluster1 2023-10-11T10:44:47.066 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ local_cluster=cluster1 2023-10-11T10:44:47.066 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval instance=1 2023-10-11T10:44:47.066 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ instance=1 2023-10-11T10:44:47.066 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-10-11T10:44:47.079 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T10:44:47.079 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T10:44:47.079 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T10:44:47.079 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-10-11T10:44:47.170 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-10-11T10:44:47.170 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd mirror status : get status for rbd mirror 2023-10-11T10:44:47.170 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:admin_socket: invalid command 2023-10-11T10:44:47.174 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-10-11T10:44:47.174 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2023-10-11T10:44:47.174 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1:2 2023-10-11T10:44:47.174 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T10:44:47.174 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance 2023-10-11T10:44:47.175 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-10-11T10:44:47.175 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local val=cluster1:2 2023-10-11T10:44:47.175 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_var_name=cluster 2023-10-11T10:44:47.175 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance_var_name=instance 2023-10-11T10:44:47.176 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ cluster=cluster1 2023-10-11T10:44:47.176 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ instance=2 2023-10-11T10:44:47.176 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' 2 = cluster1:2 ']' 2023-10-11T10:44:47.176 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval cluster=cluster1 2023-10-11T10:44:47.176 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T10:44:47.177 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval instance=2 2023-10-11T10:44:47.177 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=2 2023-10-11T10:44:47.177 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-10-11T10:44:47.177 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local local_cluster=cluster1:2 2023-10-11T10:44:47.177 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster=cluster1 2023-10-11T10:44:47.178 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance 2023-10-11T10:44:47.178 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-10-11T10:44:47.178 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local val=cluster1:2 2023-10-11T10:44:47.178 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster_var_name=local_cluster 2023-10-11T10:44:47.178 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance_var_name=instance 2023-10-11T10:44:47.179 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T10:44:47.179 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=2 2023-10-11T10:44:47.179 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ '[' 2 = cluster1:2 ']' 2023-10-11T10:44:47.179 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval local_cluster=cluster1 2023-10-11T10:44:47.179 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ local_cluster=cluster1 2023-10-11T10:44:47.180 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval instance=2 2023-10-11T10:44:47.180 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ instance=2 2023-10-11T10:44:47.180 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-10-11T10:44:47.191 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-10-11T10:44:47.191 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-10-11T10:44:47.191 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-10-11T10:44:47.192 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2023-10-11T10:44:47.281 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-10-11T10:44:47.281 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd mirror status : get status for rbd mirror 2023-10-11T10:44:47.282 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:admin_socket: invalid command 2023-10-11T10:44:47.284 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-10-11T10:44:47.284 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2023-10-11T10:44:47.284 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1:3 2023-10-11T10:44:47.285 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T10:44:47.285 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance 2023-10-11T10:44:47.285 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-10-11T10:44:47.285 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local val=cluster1:3 2023-10-11T10:44:47.285 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_var_name=cluster 2023-10-11T10:44:47.286 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance_var_name=instance 2023-10-11T10:44:47.286 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ cluster=cluster1 2023-10-11T10:44:47.286 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ instance=3 2023-10-11T10:44:47.286 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' 3 = cluster1:3 ']' 2023-10-11T10:44:47.286 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval cluster=cluster1 2023-10-11T10:44:47.287 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T10:44:47.287 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval instance=3 2023-10-11T10:44:47.287 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=3 2023-10-11T10:44:47.287 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-10-11T10:44:47.287 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local local_cluster=cluster1:3 2023-10-11T10:44:47.287 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster=cluster1 2023-10-11T10:44:47.288 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance 2023-10-11T10:44:47.288 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-10-11T10:44:47.288 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local val=cluster1:3 2023-10-11T10:44:47.288 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster_var_name=local_cluster 2023-10-11T10:44:47.288 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance_var_name=instance 2023-10-11T10:44:47.288 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T10:44:47.289 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=3 2023-10-11T10:44:47.289 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ '[' 3 = cluster1:3 ']' 2023-10-11T10:44:47.289 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval local_cluster=cluster1 2023-10-11T10:44:47.289 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ local_cluster=cluster1 2023-10-11T10:44:47.289 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval instance=3 2023-10-11T10:44:47.290 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ instance=3 2023-10-11T10:44:47.290 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-10-11T10:44:47.302 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-10-11T10:44:47.303 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-10-11T10:44:47.303 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-10-11T10:44:47.303 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2023-10-11T10:44:47.393 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-10-11T10:44:47.393 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd mirror status : get status for rbd mirror 2023-10-11T10:44:47.393 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:admin_socket: invalid command 2023-10-11T10:44:47.394 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-10-11T10:44:47.394 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sleep 1 2023-10-11T10:44:48.395 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-10-11T10:44:48.395 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ seq 0 3 2023-10-11T10:44:48.398 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-10-11T10:44:48.399 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-10-11T10:44:48.399 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1:0 2023-10-11T10:44:48.399 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T10:44:48.399 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance 2023-10-11T10:44:48.399 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-10-11T10:44:48.400 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local val=cluster1:0 2023-10-11T10:44:48.400 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_var_name=cluster 2023-10-11T10:44:48.400 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance_var_name=instance 2023-10-11T10:44:48.400 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ cluster=cluster1 2023-10-11T10:44:48.400 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ instance=0 2023-10-11T10:44:48.400 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' 0 = cluster1:0 ']' 2023-10-11T10:44:48.401 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval cluster=cluster1 2023-10-11T10:44:48.401 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T10:44:48.401 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval instance=0 2023-10-11T10:44:48.401 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=0 2023-10-11T10:44:48.401 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-10-11T10:44:48.401 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local local_cluster=cluster1:0 2023-10-11T10:44:48.402 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster=cluster1 2023-10-11T10:44:48.402 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance 2023-10-11T10:44:48.402 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-10-11T10:44:48.402 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local val=cluster1:0 2023-10-11T10:44:48.402 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster_var_name=local_cluster 2023-10-11T10:44:48.403 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance_var_name=instance 2023-10-11T10:44:48.403 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T10:44:48.403 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=0 2023-10-11T10:44:48.403 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ '[' 0 = cluster1:0 ']' 2023-10-11T10:44:48.403 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval local_cluster=cluster1 2023-10-11T10:44:48.403 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ local_cluster=cluster1 2023-10-11T10:44:48.404 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval instance=0 2023-10-11T10:44:48.404 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ instance=0 2023-10-11T10:44:48.404 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-10-11T10:44:48.420 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T10:44:48.420 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T10:44:48.420 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T10:44:48.420 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-10-11T10:44:48.517 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-10-11T10:44:48.517 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd mirror status : get status for rbd mirror 2023-10-11T10:44:48.517 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:admin_socket: invalid command 2023-10-11T10:44:48.520 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-10-11T10:44:48.520 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-10-11T10:44:48.520 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1:1 2023-10-11T10:44:48.521 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T10:44:48.521 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance 2023-10-11T10:44:48.521 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-10-11T10:44:48.521 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local val=cluster1:1 2023-10-11T10:44:48.521 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_var_name=cluster 2023-10-11T10:44:48.521 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance_var_name=instance 2023-10-11T10:44:48.522 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ cluster=cluster1 2023-10-11T10:44:48.522 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ instance=1 2023-10-11T10:44:48.522 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' 1 = cluster1:1 ']' 2023-10-11T10:44:48.522 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval cluster=cluster1 2023-10-11T10:44:48.522 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T10:44:48.522 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval instance=1 2023-10-11T10:44:48.523 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=1 2023-10-11T10:44:48.523 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-10-11T10:44:48.523 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local local_cluster=cluster1:1 2023-10-11T10:44:48.523 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster=cluster1 2023-10-11T10:44:48.523 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance 2023-10-11T10:44:48.524 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-10-11T10:44:48.524 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local val=cluster1:1 2023-10-11T10:44:48.524 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster_var_name=local_cluster 2023-10-11T10:44:48.524 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance_var_name=instance 2023-10-11T10:44:48.524 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T10:44:48.524 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=1 2023-10-11T10:44:48.525 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ '[' 1 = cluster1:1 ']' 2023-10-11T10:44:48.525 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval local_cluster=cluster1 2023-10-11T10:44:48.525 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ local_cluster=cluster1 2023-10-11T10:44:48.525 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval instance=1 2023-10-11T10:44:48.525 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ instance=1 2023-10-11T10:44:48.525 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-10-11T10:44:48.539 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T10:44:48.539 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T10:44:48.539 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T10:44:48.539 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-10-11T10:44:48.629 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-10-11T10:44:48.629 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd mirror status : get status for rbd mirror 2023-10-11T10:44:48.629 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:admin_socket: invalid command 2023-10-11T10:44:48.632 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-10-11T10:44:48.632 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2023-10-11T10:44:48.632 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1:2 2023-10-11T10:44:48.633 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T10:44:48.633 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance 2023-10-11T10:44:48.633 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-10-11T10:44:48.633 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local val=cluster1:2 2023-10-11T10:44:48.633 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_var_name=cluster 2023-10-11T10:44:48.634 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance_var_name=instance 2023-10-11T10:44:48.634 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ cluster=cluster1 2023-10-11T10:44:48.634 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ instance=2 2023-10-11T10:44:48.634 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' 2 = cluster1:2 ']' 2023-10-11T10:44:48.634 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval cluster=cluster1 2023-10-11T10:44:48.634 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T10:44:48.635 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval instance=2 2023-10-11T10:44:48.635 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=2 2023-10-11T10:44:48.635 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-10-11T10:44:48.635 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local local_cluster=cluster1:2 2023-10-11T10:44:48.635 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster=cluster1 2023-10-11T10:44:48.636 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance 2023-10-11T10:44:48.636 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-10-11T10:44:48.636 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local val=cluster1:2 2023-10-11T10:44:48.636 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster_var_name=local_cluster 2023-10-11T10:44:48.636 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance_var_name=instance 2023-10-11T10:44:48.636 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T10:44:48.637 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=2 2023-10-11T10:44:48.637 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ '[' 2 = cluster1:2 ']' 2023-10-11T10:44:48.637 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval local_cluster=cluster1 2023-10-11T10:44:48.637 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ local_cluster=cluster1 2023-10-11T10:44:48.637 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval instance=2 2023-10-11T10:44:48.637 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ instance=2 2023-10-11T10:44:48.638 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-10-11T10:44:48.650 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-10-11T10:44:48.650 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-10-11T10:44:48.651 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-10-11T10:44:48.651 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2023-10-11T10:44:48.741 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-10-11T10:44:48.741 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd mirror status : get status for rbd mirror 2023-10-11T10:44:48.741 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:admin_socket: invalid command 2023-10-11T10:44:48.744 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-10-11T10:44:48.744 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2023-10-11T10:44:48.744 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1:3 2023-10-11T10:44:48.744 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T10:44:48.744 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance 2023-10-11T10:44:48.745 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-10-11T10:44:48.745 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local val=cluster1:3 2023-10-11T10:44:48.745 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_var_name=cluster 2023-10-11T10:44:48.745 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance_var_name=instance 2023-10-11T10:44:48.745 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ cluster=cluster1 2023-10-11T10:44:48.745 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ instance=3 2023-10-11T10:44:48.746 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' 3 = cluster1:3 ']' 2023-10-11T10:44:48.746 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval cluster=cluster1 2023-10-11T10:44:48.746 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T10:44:48.746 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval instance=3 2023-10-11T10:44:48.746 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=3 2023-10-11T10:44:48.747 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-10-11T10:44:48.747 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local local_cluster=cluster1:3 2023-10-11T10:44:48.747 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster=cluster1 2023-10-11T10:44:48.747 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance 2023-10-11T10:44:48.747 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-10-11T10:44:48.748 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local val=cluster1:3 2023-10-11T10:44:48.748 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster_var_name=local_cluster 2023-10-11T10:44:48.748 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance_var_name=instance 2023-10-11T10:44:48.748 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T10:44:48.748 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=3 2023-10-11T10:44:48.748 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ '[' 3 = cluster1:3 ']' 2023-10-11T10:44:48.749 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval local_cluster=cluster1 2023-10-11T10:44:48.749 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ local_cluster=cluster1 2023-10-11T10:44:48.749 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval instance=3 2023-10-11T10:44:48.749 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ instance=3 2023-10-11T10:44:48.749 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-10-11T10:44:48.762 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-10-11T10:44:48.762 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-10-11T10:44:48.763 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-10-11T10:44:48.763 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2023-10-11T10:44:48.853 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-10-11T10:44:48.853 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd mirror status : get status for rbd mirror 2023-10-11T10:44:48.853 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:admin_socket: invalid command 2023-10-11T10:44:48.854 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-10-11T10:44:48.854 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sleep 2 2023-10-11T10:44:50.856 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-10-11T10:44:50.856 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ seq 0 3 2023-10-11T10:44:50.860 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-10-11T10:44:50.860 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-10-11T10:44:50.860 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1:0 2023-10-11T10:44:50.861 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T10:44:50.861 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance 2023-10-11T10:44:50.861 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-10-11T10:44:50.862 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local val=cluster1:0 2023-10-11T10:44:50.862 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_var_name=cluster 2023-10-11T10:44:50.862 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance_var_name=instance 2023-10-11T10:44:50.862 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ cluster=cluster1 2023-10-11T10:44:50.863 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ instance=0 2023-10-11T10:44:50.863 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' 0 = cluster1:0 ']' 2023-10-11T10:44:50.863 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval cluster=cluster1 2023-10-11T10:44:50.863 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T10:44:50.864 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval instance=0 2023-10-11T10:44:50.864 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=0 2023-10-11T10:44:50.864 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-10-11T10:44:50.864 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local local_cluster=cluster1:0 2023-10-11T10:44:50.865 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster=cluster1 2023-10-11T10:44:50.865 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance 2023-10-11T10:44:50.865 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-10-11T10:44:50.865 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local val=cluster1:0 2023-10-11T10:44:50.866 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster_var_name=local_cluster 2023-10-11T10:44:50.866 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance_var_name=instance 2023-10-11T10:44:50.866 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T10:44:50.866 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=0 2023-10-11T10:44:50.867 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ '[' 0 = cluster1:0 ']' 2023-10-11T10:44:50.867 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval local_cluster=cluster1 2023-10-11T10:44:50.867 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ local_cluster=cluster1 2023-10-11T10:44:50.867 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval instance=0 2023-10-11T10:44:50.868 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ instance=0 2023-10-11T10:44:50.868 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-10-11T10:44:50.882 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T10:44:50.883 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T10:44:50.883 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T10:44:50.883 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-10-11T10:44:50.988 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-10-11T10:44:50.988 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd mirror status : get status for rbd mirror 2023-10-11T10:44:50.988 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:admin_socket: invalid command 2023-10-11T10:44:50.992 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-10-11T10:44:50.992 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-10-11T10:44:50.992 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1:1 2023-10-11T10:44:50.992 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T10:44:50.993 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance 2023-10-11T10:44:50.993 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-10-11T10:44:50.993 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local val=cluster1:1 2023-10-11T10:44:50.993 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_var_name=cluster 2023-10-11T10:44:50.993 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance_var_name=instance 2023-10-11T10:44:50.994 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ cluster=cluster1 2023-10-11T10:44:50.994 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ instance=1 2023-10-11T10:44:50.994 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' 1 = cluster1:1 ']' 2023-10-11T10:44:50.994 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval cluster=cluster1 2023-10-11T10:44:50.995 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T10:44:50.995 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval instance=1 2023-10-11T10:44:50.995 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=1 2023-10-11T10:44:50.995 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-10-11T10:44:50.995 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local local_cluster=cluster1:1 2023-10-11T10:44:50.996 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster=cluster1 2023-10-11T10:44:50.996 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance 2023-10-11T10:44:50.996 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-10-11T10:44:50.996 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local val=cluster1:1 2023-10-11T10:44:50.997 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster_var_name=local_cluster 2023-10-11T10:44:50.997 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance_var_name=instance 2023-10-11T10:44:50.997 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T10:44:50.997 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=1 2023-10-11T10:44:50.998 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ '[' 1 = cluster1:1 ']' 2023-10-11T10:44:50.998 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval local_cluster=cluster1 2023-10-11T10:44:50.999 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ local_cluster=cluster1 2023-10-11T10:44:50.999 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval instance=1 2023-10-11T10:44:50.999 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ instance=1 2023-10-11T10:44:50.999 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-10-11T10:44:51.011 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T10:44:51.011 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T10:44:51.011 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T10:44:51.011 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-10-11T10:44:51.103 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-10-11T10:44:51.103 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd mirror status : get status for rbd mirror 2023-10-11T10:44:51.103 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:admin_socket: invalid command 2023-10-11T10:44:51.105 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-10-11T10:44:51.105 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2023-10-11T10:44:51.106 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1:2 2023-10-11T10:44:51.106 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T10:44:51.106 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance 2023-10-11T10:44:51.106 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-10-11T10:44:51.107 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local val=cluster1:2 2023-10-11T10:44:51.107 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_var_name=cluster 2023-10-11T10:44:51.107 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance_var_name=instance 2023-10-11T10:44:51.107 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ cluster=cluster1 2023-10-11T10:44:51.107 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ instance=2 2023-10-11T10:44:51.108 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' 2 = cluster1:2 ']' 2023-10-11T10:44:51.108 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval cluster=cluster1 2023-10-11T10:44:51.108 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T10:44:51.108 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval instance=2 2023-10-11T10:44:51.108 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=2 2023-10-11T10:44:51.109 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-10-11T10:44:51.110 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local local_cluster=cluster1:2 2023-10-11T10:44:51.110 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster=cluster1 2023-10-11T10:44:51.111 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance 2023-10-11T10:44:51.111 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-10-11T10:44:51.111 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local val=cluster1:2 2023-10-11T10:44:51.111 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster_var_name=local_cluster 2023-10-11T10:44:51.112 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance_var_name=instance 2023-10-11T10:44:51.112 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T10:44:51.112 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=2 2023-10-11T10:44:51.112 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ '[' 2 = cluster1:2 ']' 2023-10-11T10:44:51.112 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval local_cluster=cluster1 2023-10-11T10:44:51.113 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ local_cluster=cluster1 2023-10-11T10:44:51.113 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval instance=2 2023-10-11T10:44:51.113 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ instance=2 2023-10-11T10:44:51.113 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-10-11T10:44:51.124 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-10-11T10:44:51.124 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-10-11T10:44:51.125 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-10-11T10:44:51.125 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2023-10-11T10:44:51.214 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-10-11T10:44:51.214 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd mirror status : get status for rbd mirror 2023-10-11T10:44:51.215 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:admin_socket: invalid command 2023-10-11T10:44:51.217 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-10-11T10:44:51.217 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2023-10-11T10:44:51.217 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1:3 2023-10-11T10:44:51.217 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T10:44:51.218 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance 2023-10-11T10:44:51.218 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-10-11T10:44:51.218 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local val=cluster1:3 2023-10-11T10:44:51.218 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_var_name=cluster 2023-10-11T10:44:51.219 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance_var_name=instance 2023-10-11T10:44:51.219 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ cluster=cluster1 2023-10-11T10:44:51.219 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ instance=3 2023-10-11T10:44:51.219 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' 3 = cluster1:3 ']' 2023-10-11T10:44:51.219 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval cluster=cluster1 2023-10-11T10:44:51.220 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T10:44:51.220 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval instance=3 2023-10-11T10:44:51.220 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=3 2023-10-11T10:44:51.220 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-10-11T10:44:51.220 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local local_cluster=cluster1:3 2023-10-11T10:44:51.222 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster=cluster1 2023-10-11T10:44:51.222 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance 2023-10-11T10:44:51.223 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-10-11T10:44:51.223 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local val=cluster1:3 2023-10-11T10:44:51.223 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster_var_name=local_cluster 2023-10-11T10:44:51.223 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance_var_name=instance 2023-10-11T10:44:51.224 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T10:44:51.224 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=3 2023-10-11T10:44:51.224 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ '[' 3 = cluster1:3 ']' 2023-10-11T10:44:51.224 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval local_cluster=cluster1 2023-10-11T10:44:51.224 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ local_cluster=cluster1 2023-10-11T10:44:51.225 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval instance=3 2023-10-11T10:44:51.225 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ instance=3 2023-10-11T10:44:51.225 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-10-11T10:44:51.235 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-10-11T10:44:51.236 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-10-11T10:44:51.236 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-10-11T10:44:51.236 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2023-10-11T10:44:51.326 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-10-11T10:44:51.326 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd mirror status : get status for rbd mirror 2023-10-11T10:44:51.326 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:admin_socket: invalid command 2023-10-11T10:44:51.327 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-10-11T10:44:51.327 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sleep 4 2023-10-11T10:44:55.328 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-10-11T10:44:55.328 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ seq 0 3 2023-10-11T10:44:55.332 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-10-11T10:44:55.332 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-10-11T10:44:55.332 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1:0 2023-10-11T10:44:55.332 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T10:44:55.332 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance 2023-10-11T10:44:55.333 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-10-11T10:44:55.333 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local val=cluster1:0 2023-10-11T10:44:55.333 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_var_name=cluster 2023-10-11T10:44:55.333 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance_var_name=instance 2023-10-11T10:44:55.333 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ cluster=cluster1 2023-10-11T10:44:55.334 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ instance=0 2023-10-11T10:44:55.334 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' 0 = cluster1:0 ']' 2023-10-11T10:44:55.334 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval cluster=cluster1 2023-10-11T10:44:55.334 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T10:44:55.334 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval instance=0 2023-10-11T10:44:55.335 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=0 2023-10-11T10:44:55.335 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-10-11T10:44:55.335 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local local_cluster=cluster1:0 2023-10-11T10:44:55.335 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster=cluster1 2023-10-11T10:44:55.335 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance 2023-10-11T10:44:55.336 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-10-11T10:44:55.336 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local val=cluster1:0 2023-10-11T10:44:55.336 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster_var_name=local_cluster 2023-10-11T10:44:55.336 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance_var_name=instance 2023-10-11T10:44:55.336 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T10:44:55.337 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=0 2023-10-11T10:44:55.337 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ '[' 0 = cluster1:0 ']' 2023-10-11T10:44:55.337 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval local_cluster=cluster1 2023-10-11T10:44:55.337 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ local_cluster=cluster1 2023-10-11T10:44:55.337 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval instance=0 2023-10-11T10:44:55.338 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ instance=0 2023-10-11T10:44:55.338 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-10-11T10:44:55.353 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T10:44:55.354 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T10:44:55.354 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T10:44:55.354 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-10-11T10:44:55.453 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-10-11T10:44:55.454 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd mirror status : get status for rbd mirror 2023-10-11T10:44:55.454 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:admin_socket: invalid command 2023-10-11T10:44:55.456 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-10-11T10:44:55.456 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-10-11T10:44:55.456 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1:1 2023-10-11T10:44:55.456 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T10:44:55.457 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance 2023-10-11T10:44:55.457 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-10-11T10:44:55.457 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local val=cluster1:1 2023-10-11T10:44:55.457 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_var_name=cluster 2023-10-11T10:44:55.457 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance_var_name=instance 2023-10-11T10:44:55.458 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ cluster=cluster1 2023-10-11T10:44:55.458 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ instance=1 2023-10-11T10:44:55.458 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' 1 = cluster1:1 ']' 2023-10-11T10:44:55.458 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval cluster=cluster1 2023-10-11T10:44:55.458 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T10:44:55.459 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval instance=1 2023-10-11T10:44:55.459 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=1 2023-10-11T10:44:55.459 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-10-11T10:44:55.461 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local local_cluster=cluster1:1 2023-10-11T10:44:55.461 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster=cluster1 2023-10-11T10:44:55.461 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance 2023-10-11T10:44:55.461 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-10-11T10:44:55.461 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local val=cluster1:1 2023-10-11T10:44:55.462 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster_var_name=local_cluster 2023-10-11T10:44:55.462 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance_var_name=instance 2023-10-11T10:44:55.462 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T10:44:55.462 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=1 2023-10-11T10:44:55.462 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ '[' 1 = cluster1:1 ']' 2023-10-11T10:44:55.463 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval local_cluster=cluster1 2023-10-11T10:44:55.463 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ local_cluster=cluster1 2023-10-11T10:44:55.463 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval instance=1 2023-10-11T10:44:55.463 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ instance=1 2023-10-11T10:44:55.463 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-10-11T10:44:55.476 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T10:44:55.476 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T10:44:55.476 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T10:44:55.477 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-10-11T10:44:55.579 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ return 0 2023-10-11T10:44:55.580 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ status_result=' "state": "Replaying"' 2023-10-11T10:44:55.580 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ echo ' "state": "Replaying"' 2023-10-11T10:44:55.580 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ grep -i Replaying 2023-10-11T10:44:55.581 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: "state": "Replaying" 2023-10-11T10:44:55.582 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ current_state=started 2023-10-11T10:44:55.582 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test started = started 2023-10-11T10:44:55.582 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T10:44:55.583 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ clean_snap_name= 2023-10-11T10:44:55.583 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ seq 1 10 2023-10-11T10:44:55.583 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 10` 2023-10-11T10:44:55.583 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ stress_write_image cluster2 mirror test 2023-10-11T10:44:55.583 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:44:55.584 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:44:55.584 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:44:55.584 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-10-11T10:44:55.585 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local duration=6 2023-10-11T10:44:55.585 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ set +e 2023-10-11T10:44:55.585 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2023-10-11T10:44:55.781 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:100 IOs, 25900032 bytes 2023-10-11T10:44:55.939 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:200 IOs, 67082240 bytes 2023-10-11T10:44:56.056 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:300 IOs, 94115840 bytes 2023-10-11T10:44:56.226 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:400 IOs, 153409536 bytes 2023-10-11T10:44:56.348 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:500 IOs, 189321216 bytes 2023-10-11T10:44:56.496 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:600 IOs, 232230912 bytes 2023-10-11T10:44:56.730 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:700 IOs, 328318976 bytes 2023-10-11T10:44:56.880 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:800 IOs, 378719232 bytes 2023-10-11T10:44:57.235 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:900 IOs, 440475648 bytes 2023-10-11T10:44:57.517 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1000 IOs, 553658368 bytes 2023-10-11T10:44:57.728 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1100 IOs, 625222656 bytes 2023-10-11T10:44:57.907 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1200 IOs, 683082752 bytes 2023-10-11T10:44:58.056 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1300 IOs, 726211584 bytes 2023-10-11T10:44:58.150 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1400 IOs, 740458496 bytes 2023-10-11T10:44:58.275 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1500 IOs, 780572672 bytes 2023-10-11T10:44:58.377 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1600 IOs, 795608064 bytes 2023-10-11T10:44:58.479 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1700 IOs, 814277632 bytes 2023-10-11T10:44:58.594 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1800 IOs, 846215168 bytes 2023-10-11T10:44:58.744 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1900 IOs, 879579136 bytes 2023-10-11T10:44:58.870 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2000 IOs, 914192384 bytes 2023-10-11T10:44:59.027 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2100 IOs, 961242112 bytes 2023-10-11T10:44:59.153 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2200 IOs, 986992640 bytes 2023-10-11T10:44:59.290 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2300 IOs, 1033832448 bytes 2023-10-11T10:44:59.477 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2400 IOs, 1072508928 bytes 2023-10-11T10:44:59.583 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2500 IOs, 1101006848 bytes 2023-10-11T10:44:59.662 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2600 IOs, 1114381312 bytes 2023-10-11T10:44:59.779 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2700 IOs, 1154234368 bytes 2023-10-11T10:44:59.967 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2800 IOs, 1214879744 bytes 2023-10-11T10:45:00.110 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2900 IOs, 1260042240 bytes 2023-10-11T10:45:00.259 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3000 IOs, 1303318528 bytes 2023-10-11T10:45:00.374 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3100 IOs, 1330697216 bytes 2023-10-11T10:45:00.522 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3200 IOs, 1383827456 bytes 2023-10-11T10:45:00.674 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3300 IOs, 1428331520 bytes 2023-10-11T10:45:00.958 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3400 IOs, 1540162560 bytes 2023-10-11T10:45:01.048 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3500 IOs, 1551882240 bytes 2023-10-11T10:45:01.172 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3600 IOs, 1580447744 bytes 2023-10-11T10:45:01.304 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3700 IOs, 1607871488 bytes 2023-10-11T10:45:01.564 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3800 IOs, 1698057216 bytes 2023-10-11T10:45:01.590 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ error_code=124 2023-10-11T10:45:01.590 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ set -e 2023-10-11T10:45:01.590 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' 124 -eq 124 ']' 2023-10-11T10:45:01.591 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T10:45:01.591 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2023-10-11T10:45:01.591 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T10:45:01.591 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:45:01.592 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:45:01.592 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local state_pattern=up+replaying 2023-10-11T10:45:01.592 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local description_pattern= 2023-10-11T10:45:01.592 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local service_pattern= 2023-10-11T10:45:01.592 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-10-11T10:45:01.593 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 1 2023-10-11T10:45:02.591 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2023-10-11T10:45:02.592 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T10:45:02.592 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:45:02.592 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:45:02.592 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local state_pattern=up+replaying 2023-10-11T10:45:02.593 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local description_pattern= 2023-10-11T10:45:02.593 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local service_pattern= 2023-10-11T10:45:02.593 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ mkfname cluster1-mirror-test.mirror_status 2023-10-11T10:45:02.593 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ echo cluster1-mirror-test.mirror_status 2023-10-11T10:45:02.593 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -e 's|[/ ]|_|g' 2023-10-11T10:45:02.594 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local status_log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.mirror_status 2023-10-11T10:45:02.594 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ CEPH_ARGS= 2023-10-11T10:45:02.594 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2023-10-11T10:45:02.595 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ tee /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.mirror_status 2023-10-11T10:45:02.640 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:test: 2023-10-11T10:45:02.640 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: global_id: 3dd93db4-c9cc-462c-8b49-007eba9a90d2 2023-10-11T10:45:02.640 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: state: down+unknown 2023-10-11T10:45:02.640 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: description: status not found 2023-10-11T10:45:02.641 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: last_update: 2023-10-11T10:45:02.641 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: peer_sites: 2023-10-11T10:45:02.641 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: name: cluster2 2023-10-11T10:45:02.641 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: state: down+unknown 2023-10-11T10:45:02.641 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: description: status not found 2023-10-11T10:45:02.641 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: last_update: 2023-10-11T10:45:02.645 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.mirror_status 2023-10-11T10:45:02.646 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 1 2023-10-11T10:45:02.647 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-10-11T10:45:02.647 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 2 2023-10-11T10:45:04.648 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2023-10-11T10:45:04.649 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T10:45:04.649 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:45:04.649 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:45:04.649 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local state_pattern=up+replaying 2023-10-11T10:45:04.649 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local description_pattern= 2023-10-11T10:45:04.650 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local service_pattern= 2023-10-11T10:45:04.650 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ mkfname cluster1-mirror-test.mirror_status 2023-10-11T10:45:04.650 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ echo cluster1-mirror-test.mirror_status 2023-10-11T10:45:04.650 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -e 's|[/ ]|_|g' 2023-10-11T10:45:04.651 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local status_log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.mirror_status 2023-10-11T10:45:04.651 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ CEPH_ARGS= 2023-10-11T10:45:04.651 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2023-10-11T10:45:04.651 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ tee /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.mirror_status 2023-10-11T10:45:04.696 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:test: 2023-10-11T10:45:04.697 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: global_id: 3dd93db4-c9cc-462c-8b49-007eba9a90d2 2023-10-11T10:45:04.697 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: state: down+unknown 2023-10-11T10:45:04.697 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: description: status not found 2023-10-11T10:45:04.697 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: last_update: 2023-10-11T10:45:04.697 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: peer_sites: 2023-10-11T10:45:04.697 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: name: cluster2 2023-10-11T10:45:04.698 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: state: down+unknown 2023-10-11T10:45:04.698 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: description: status not found 2023-10-11T10:45:04.698 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: last_update: 2023-10-11T10:45:04.701 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.mirror_status 2023-10-11T10:45:04.705 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 1 2023-10-11T10:45:04.705 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-10-11T10:45:04.706 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 4 2023-10-11T10:45:08.704 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2023-10-11T10:45:08.705 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T10:45:08.705 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:45:08.705 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:45:08.705 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local state_pattern=up+replaying 2023-10-11T10:45:08.706 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local description_pattern= 2023-10-11T10:45:08.706 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local service_pattern= 2023-10-11T10:45:08.706 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ mkfname cluster1-mirror-test.mirror_status 2023-10-11T10:45:08.706 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ echo cluster1-mirror-test.mirror_status 2023-10-11T10:45:08.706 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -e 's|[/ ]|_|g' 2023-10-11T10:45:08.707 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local status_log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.mirror_status 2023-10-11T10:45:08.707 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ CEPH_ARGS= 2023-10-11T10:45:08.707 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2023-10-11T10:45:08.708 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ tee /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.mirror_status 2023-10-11T10:45:08.754 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:test: 2023-10-11T10:45:08.754 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: global_id: 3dd93db4-c9cc-462c-8b49-007eba9a90d2 2023-10-11T10:45:08.754 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: state: down+unknown 2023-10-11T10:45:08.754 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: description: status not found 2023-10-11T10:45:08.755 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: last_update: 2023-10-11T10:45:08.755 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: peer_sites: 2023-10-11T10:45:08.755 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: name: cluster2 2023-10-11T10:45:08.755 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: state: down+unknown 2023-10-11T10:45:08.755 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: description: status not found 2023-10-11T10:45:08.755 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: last_update: 2023-10-11T10:45:08.759 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.mirror_status 2023-10-11T10:45:08.760 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 1 2023-10-11T10:45:08.760 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-10-11T10:45:08.760 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 8 2023-10-11T10:45:16.762 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2023-10-11T10:45:16.762 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T10:45:16.762 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:45:16.763 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:45:16.763 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local state_pattern=up+replaying 2023-10-11T10:45:16.763 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local description_pattern= 2023-10-11T10:45:16.763 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local service_pattern= 2023-10-11T10:45:16.763 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ mkfname cluster1-mirror-test.mirror_status 2023-10-11T10:45:16.764 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ echo cluster1-mirror-test.mirror_status 2023-10-11T10:45:16.764 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -e 's|[/ ]|_|g' 2023-10-11T10:45:16.764 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local status_log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.mirror_status 2023-10-11T10:45:16.765 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ CEPH_ARGS= 2023-10-11T10:45:16.765 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2023-10-11T10:45:16.765 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ tee /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.mirror_status 2023-10-11T10:45:16.811 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:test: 2023-10-11T10:45:16.811 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: global_id: 3dd93db4-c9cc-462c-8b49-007eba9a90d2 2023-10-11T10:45:16.811 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: state: down+unknown 2023-10-11T10:45:16.812 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: description: status not found 2023-10-11T10:45:16.812 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: last_update: 2023-10-11T10:45:16.812 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: peer_sites: 2023-10-11T10:45:16.812 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: name: cluster2 2023-10-11T10:45:16.812 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: state: down+unknown 2023-10-11T10:45:16.812 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: description: status not found 2023-10-11T10:45:16.813 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: last_update: 2023-10-11T10:45:16.816 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.mirror_status 2023-10-11T10:45:16.817 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 1 2023-10-11T10:45:16.817 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-10-11T10:45:16.818 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 8 2023-10-11T10:45:24.819 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2023-10-11T10:45:24.819 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T10:45:24.819 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:45:24.820 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:45:24.820 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local state_pattern=up+replaying 2023-10-11T10:45:24.820 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local description_pattern= 2023-10-11T10:45:24.820 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local service_pattern= 2023-10-11T10:45:24.820 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ mkfname cluster1-mirror-test.mirror_status 2023-10-11T10:45:24.821 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ echo cluster1-mirror-test.mirror_status 2023-10-11T10:45:24.822 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -e 's|[/ ]|_|g' 2023-10-11T10:45:24.822 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local status_log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.mirror_status 2023-10-11T10:45:24.822 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ CEPH_ARGS= 2023-10-11T10:45:24.822 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2023-10-11T10:45:24.822 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ tee /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.mirror_status 2023-10-11T10:45:24.870 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:test: 2023-10-11T10:45:24.870 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: global_id: 3dd93db4-c9cc-462c-8b49-007eba9a90d2 2023-10-11T10:45:24.870 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: state: up+replaying 2023-10-11T10:45:24.871 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1697021085,"remote_snapshot_timestamp":1697021085,"replay_state":"idle"} 2023-10-11T10:45:24.871 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: service: mirror.1 on smithi111 2023-10-11T10:45:24.871 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: last_update: 2023-10-11 10:45:21 2023-10-11T10:45:24.871 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: peer_sites: 2023-10-11T10:45:24.871 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: name: cluster2 2023-10-11T10:45:24.872 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: state: up+stopped 2023-10-11T10:45:24.872 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: description: local image is primary 2023-10-11T10:45:24.872 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: last_update: 2023-10-11 10:45:21 2023-10-11T10:45:24.875 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.mirror_status 2023-10-11T10:45:24.876 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: state: up+replaying 2023-10-11T10:45:24.877 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ grep '^ description: .*' /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.mirror_status 2023-10-11T10:45:24.877 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1697021085,"remote_snapshot_timestamp":1697021085,"replay_state":"idle"} 2023-10-11T10:45:24.878 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n '' ']' 2023-10-11T10:45:24.878 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ echo up+replaying 2023-10-11T10:45:24.878 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ grep '^up+' 2023-10-11T10:45:24.879 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:up+replaying 2023-10-11T10:45:24.879 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ grep 'service: *mirror..* on ' /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.mirror_status 2023-10-11T10:45:24.880 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: service: mirror.1 on smithi111 2023-10-11T10:45:24.880 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.mirror_status 2023-10-11T10:45:24.881 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local 'last_update=2023-10-11 10:45:21' 2023-10-11T10:45:24.882 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2023-10-11 10:45:21' 2023-10-11T10:45:24.882 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T10:45:24.882 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:45:24.882 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:45:24.882 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local state_pattern=up+replaying 2023-10-11T10:45:24.883 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local 'prev_last_update=2023-10-11 10:45:21' 2023-10-11T10:45:24.883 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ mkfname cluster1-mirror.mirror_status 2023-10-11T10:45:24.883 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ echo cluster1-mirror.mirror_status 2023-10-11T10:45:24.883 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -e 's|[/ ]|_|g' 2023-10-11T10:45:24.884 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local status_log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror.mirror_status 2023-10-11T10:45:24.884 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2023-10-11T10:45:24.920 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd: failed to query services: (13) Permission denied 2023-10-11T10:45:24.940 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local last_update state 2023-10-11T10:45:24.941 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2023-10-11T10:45:24.989 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ last_update='2023-10-11 10:45:21' 2023-10-11T10:45:24.989 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2023-10-11T10:45:24.991 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ state=up+replaying 2023-10-11T10:45:24.991 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ echo up+replaying 2023-10-11T10:45:24.991 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ grep up+replaying 2023-10-11T10:45:24.992 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:up+replaying 2023-10-11T10:45:24.993 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T10:45:24.993 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T10:45:24.993 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ snap_name=snap1 2023-10-11T10:45:24.993 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ create_snap cluster2 mirror test snap1 2023-10-11T10:45:24.993 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:45:24.994 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:45:24.994 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:45:24.994 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap1 2023-10-11T10:45:24.994 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap1 --debug-rbd=20 --debug-journaler=20 2023-10-11T10:45:32.825 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_replay_started cluster1 mirror test 2023-10-11T10:45:32.826 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T10:45:32.826 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:45:32.826 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:45:32.826 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2023-10-11T10:45:32.827 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T10:45:32.827 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:45:32.827 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:45:32.827 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local state=started 2023-10-11T10:45:32.827 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T10:45:32.827 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-10-11T10:45:32.828 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 1 2023-10-11T10:45:33.826 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test_image_replay_state cluster1 mirror test started 2023-10-11T10:45:33.827 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T10:45:33.827 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:45:33.827 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:45:33.827 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local test_state=started 2023-10-11T10:45:33.828 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local status_result 2023-10-11T10:45:33.828 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_state=stopped 2023-10-11T10:45:33.829 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2023-10-11T10:45:33.829 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ grep -i state 2023-10-11T10:45:33.829 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_instance=cluster1 2023-10-11T10:45:33.829 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T10:45:33.830 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T10:45:33.830 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance=cluster1 2023-10-11T10:45:33.830 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local loop_instance 2023-10-11T10:45:33.830 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-10-11T10:45:33.830 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sleep 0 2023-10-11T10:45:33.831 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-10-11T10:45:33.831 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ seq 0 3 2023-10-11T10:45:33.832 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-10-11T10:45:33.832 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-10-11T10:45:33.833 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1:0 2023-10-11T10:45:33.833 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T10:45:33.833 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance 2023-10-11T10:45:33.833 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-10-11T10:45:33.833 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local val=cluster1:0 2023-10-11T10:45:33.834 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_var_name=cluster 2023-10-11T10:45:33.834 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance_var_name=instance 2023-10-11T10:45:33.834 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ cluster=cluster1 2023-10-11T10:45:33.834 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ instance=0 2023-10-11T10:45:33.834 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' 0 = cluster1:0 ']' 2023-10-11T10:45:33.834 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval cluster=cluster1 2023-10-11T10:45:33.835 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T10:45:33.835 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval instance=0 2023-10-11T10:45:33.835 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=0 2023-10-11T10:45:33.835 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-10-11T10:45:33.835 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local local_cluster=cluster1:0 2023-10-11T10:45:33.836 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster=cluster1 2023-10-11T10:45:33.836 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance 2023-10-11T10:45:33.837 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-10-11T10:45:33.837 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local val=cluster1:0 2023-10-11T10:45:33.837 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster_var_name=local_cluster 2023-10-11T10:45:33.837 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance_var_name=instance 2023-10-11T10:45:33.838 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T10:45:33.838 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=0 2023-10-11T10:45:33.838 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ '[' 0 = cluster1:0 ']' 2023-10-11T10:45:33.838 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval local_cluster=cluster1 2023-10-11T10:45:33.838 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ local_cluster=cluster1 2023-10-11T10:45:33.839 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval instance=0 2023-10-11T10:45:33.839 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ instance=0 2023-10-11T10:45:33.839 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-10-11T10:45:33.853 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T10:45:33.853 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T10:45:33.853 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T10:45:33.853 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-10-11T10:45:33.954 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-10-11T10:45:33.954 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd mirror status : get status for rbd mirror 2023-10-11T10:45:33.954 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:admin_socket: invalid command 2023-10-11T10:45:33.955 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-10-11T10:45:33.956 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-10-11T10:45:33.956 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1:1 2023-10-11T10:45:33.956 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T10:45:33.956 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance 2023-10-11T10:45:33.956 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-10-11T10:45:33.957 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local val=cluster1:1 2023-10-11T10:45:33.957 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_var_name=cluster 2023-10-11T10:45:33.957 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance_var_name=instance 2023-10-11T10:45:33.957 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ cluster=cluster1 2023-10-11T10:45:33.957 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ instance=1 2023-10-11T10:45:33.958 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' 1 = cluster1:1 ']' 2023-10-11T10:45:33.958 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval cluster=cluster1 2023-10-11T10:45:33.959 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T10:45:33.959 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval instance=1 2023-10-11T10:45:33.960 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=1 2023-10-11T10:45:33.960 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-10-11T10:45:33.960 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local local_cluster=cluster1:1 2023-10-11T10:45:33.960 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster=cluster1 2023-10-11T10:45:33.960 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance 2023-10-11T10:45:33.961 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-10-11T10:45:33.961 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local val=cluster1:1 2023-10-11T10:45:33.961 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster_var_name=local_cluster 2023-10-11T10:45:33.961 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance_var_name=instance 2023-10-11T10:45:33.961 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T10:45:33.961 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=1 2023-10-11T10:45:33.962 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ '[' 1 = cluster1:1 ']' 2023-10-11T10:45:33.962 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval local_cluster=cluster1 2023-10-11T10:45:33.962 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ local_cluster=cluster1 2023-10-11T10:45:33.962 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval instance=1 2023-10-11T10:45:33.962 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ instance=1 2023-10-11T10:45:33.962 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-10-11T10:45:33.975 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T10:45:33.976 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T10:45:33.976 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T10:45:33.976 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-10-11T10:45:34.079 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ return 0 2023-10-11T10:45:34.080 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ status_result=' "state": "Replaying"' 2023-10-11T10:45:34.080 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ echo ' "state": "Replaying"' 2023-10-11T10:45:34.080 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ grep -i Replaying 2023-10-11T10:45:34.081 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: "state": "Replaying" 2023-10-11T10:45:34.082 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ current_state=started 2023-10-11T10:45:34.082 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test started = started 2023-10-11T10:45:34.082 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T10:45:34.083 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2023-10-11T10:45:34.083 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local local_cluster=cluster1 2023-10-11T10:45:34.083 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:45:34.083 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:45:34.083 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:45:34.083 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' snapshot = journal ']' 2023-10-11T10:45:34.084 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' snapshot = snapshot ']' 2023-10-11T10:45:34.084 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2023-10-11T10:45:34.084 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local local_cluster=cluster1 2023-10-11T10:45:34.084 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:45:34.084 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:45:34.084 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:45:34.086 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ mkfname cluster2-mirror-test.status 2023-10-11T10:45:34.086 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ echo cluster2-mirror-test.status 2023-10-11T10:45:34.086 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -e 's|[/ ]|_|g' 2023-10-11T10:45:34.086 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local status_log=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.status 2023-10-11T10:45:34.087 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ mkfname cluster1-mirror-test.status 2023-10-11T10:45:34.087 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ echo cluster1-mirror-test.status 2023-10-11T10:45:34.087 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -e 's|[/ ]|_|g' 2023-10-11T10:45:34.087 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local local_status_log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.status 2023-10-11T10:45:34.087 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ mirror_image_snapshot cluster2 mirror test 2023-10-11T10:45:34.088 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:45:34.088 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:45:34.088 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:45:34.088 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2023-10-11T10:45:34.819 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:Snapshot ID: 6 2023-10-11T10:45:34.827 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.status 2023-10-11T10:45:34.827 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:45:34.827 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:45:34.828 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:45:34.828 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.status 2023-10-11T10:45:34.828 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2023-10-11T10:45:34.828 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T10:45:34.870 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-10-11T10:45:34.871 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snapshot_id=6 2023-10-11T10:45:34.872 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T10:45:34.872 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T10:45:34.872 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.2 2023-10-11T10:45:35.073 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.status 2023-10-11T10:45:35.073 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T10:45:35.073 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:45:35.074 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:45:35.074 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.status 2023-10-11T10:45:35.074 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-10-11T10:45:35.074 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T10:45:35.118 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T10:45:35.119 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id=4 2023-10-11T10:45:35.119 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 6 = 4 2023-10-11T10:45:35.120 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T10:45:35.120 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.4 2023-10-11T10:45:35.521 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.status 2023-10-11T10:45:35.521 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T10:45:35.521 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:45:35.521 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:45:35.522 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.status 2023-10-11T10:45:35.522 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-10-11T10:45:35.522 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T10:45:35.567 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T10:45:35.569 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id=4 2023-10-11T10:45:35.569 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 6 = 4 2023-10-11T10:45:35.569 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T10:45:35.569 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.8 2023-10-11T10:45:36.370 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.status 2023-10-11T10:45:36.370 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T10:45:36.370 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:45:36.371 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:45:36.371 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.status 2023-10-11T10:45:36.371 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-10-11T10:45:36.371 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T10:45:36.418 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T10:45:36.418 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T10:45:36.419 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:-:1.1: Document is empty 2023-10-11T10:45:36.420 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: 2023-10-11T10:45:36.420 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:^ 2023-10-11T10:45:36.421 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id= 2023-10-11T10:45:36.421 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 6 = '' 2023-10-11T10:45:36.421 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T10:45:36.421 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 1.6 2023-10-11T10:45:38.021 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.status 2023-10-11T10:45:38.022 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T10:45:38.022 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:45:38.022 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:45:38.022 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.status 2023-10-11T10:45:38.023 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-10-11T10:45:38.023 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T10:45:38.068 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T10:45:38.071 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id=6 2023-10-11T10:45:38.071 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 6 = 6 2023-10-11T10:45:38.071 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T10:45:38.071 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_snap_present cluster1 mirror test snap1 2023-10-11T10:45:38.072 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T10:45:38.072 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:45:38.072 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:45:38.072 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap1 2023-10-11T10:45:38.072 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T10:45:38.073 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-10-11T10:45:38.073 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 1 2023-10-11T10:45:38.493 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.5812001696747424) > weight (0.25) 2023-10-11T10:45:38.493 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.8190967828174539) > weight (0.25) 2023-10-11T10:45:38.494 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.9693723797755494) > weight (0.25) 2023-10-11T10:45:38.494 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.3435806332766943) > weight (0.25) 2023-10-11T10:45:38.494 INFO:tasks.rbd_mirror_thrash:waiting for 61 secs before thrashing 2023-10-11T10:45:39.072 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 info mirror/test@snap1 2023-10-11T10:45:39.119 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:rbd image 'test': 2023-10-11T10:45:39.119 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: size 512 MiB in 128 objects 2023-10-11T10:45:39.119 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: order 22 (4 MiB objects) 2023-10-11T10:45:39.120 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: snapshot_count: 2 2023-10-11T10:45:39.120 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: id: 10afefb6b972 2023-10-11T10:45:39.120 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: block_name_prefix: rbd_data.10afefb6b972 2023-10-11T10:45:39.120 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: format: 2 2023-10-11T10:45:39.120 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-10-11T10:45:39.121 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: op_features: 2023-10-11T10:45:39.121 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: flags: 2023-10-11T10:45:39.121 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: create_timestamp: Wed Oct 11 10:44:53 2023 2023-10-11T10:45:39.121 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: access_timestamp: Wed Oct 11 10:44:53 2023 2023-10-11T10:45:39.121 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: modify_timestamp: Wed Oct 11 10:44:53 2023 2023-10-11T10:45:39.121 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: protected: False 2023-10-11T10:45:39.122 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring state: enabled 2023-10-11T10:45:39.122 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring mode: snapshot 2023-10-11T10:45:39.122 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring global id: 3dd93db4-c9cc-462c-8b49-007eba9a90d2 2023-10-11T10:45:39.122 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring primary: false 2023-10-11T10:45:39.124 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T10:45:39.124 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n '' ']' 2023-10-11T10:45:39.124 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ compare_image_snaps mirror test snap1 2023-10-11T10:45:39.124 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:45:39.124 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:45:39.125 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap1 2023-10-11T10:45:39.125 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local ret=0 2023-10-11T10:45:39.125 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local rmt_export=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export 2023-10-11T10:45:39.125 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local loc_export=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:45:39.125 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rm -f /tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:45:39.126 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 -p mirror export test@snap1 /tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export 2023-10-11T10:45:39.700 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-10-11T10:45:39.708 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 -p mirror export test@snap1 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:45:40.415 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-10-11T10:45:40.424 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ cmp /tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:45:40.636 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rm -f /tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:45:40.775 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T10:45:40.776 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ clean_snap_name=snap1-clean 2023-10-11T10:45:40.776 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ create_snap cluster2 mirror test snap1-clean 2023-10-11T10:45:40.776 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:45:40.777 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:45:40.777 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:45:40.777 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap1-clean 2023-10-11T10:45:40.777 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap1-clean --debug-rbd=20 --debug-journaler=20 2023-10-11T10:45:41.831 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 10` 2023-10-11T10:45:41.832 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ stress_write_image cluster2 mirror test 2023-10-11T10:45:41.832 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:45:41.832 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:45:41.832 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:45:41.832 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-10-11T10:45:41.833 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local duration=5 2023-10-11T10:45:41.833 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ set +e 2023-10-11T10:45:41.833 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2023-10-11T10:45:42.040 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:100 IOs, 9093120 bytes 2023-10-11T10:45:42.303 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:200 IOs, 97688576 bytes 2023-10-11T10:45:42.449 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:300 IOs, 134855680 bytes 2023-10-11T10:45:42.571 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:400 IOs, 169383936 bytes 2023-10-11T10:45:42.698 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:500 IOs, 215651328 bytes 2023-10-11T10:45:42.824 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:600 IOs, 243100672 bytes 2023-10-11T10:45:43.023 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:700 IOs, 258126848 bytes 2023-10-11T10:45:43.164 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:800 IOs, 298779648 bytes 2023-10-11T10:45:43.286 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:900 IOs, 326912000 bytes 2023-10-11T10:45:43.403 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1000 IOs, 363949056 bytes 2023-10-11T10:45:43.582 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1100 IOs, 426017792 bytes 2023-10-11T10:45:43.720 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1200 IOs, 481328128 bytes 2023-10-11T10:45:43.813 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1300 IOs, 484885504 bytes 2023-10-11T10:45:43.974 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1400 IOs, 528379904 bytes 2023-10-11T10:45:44.131 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1500 IOs, 586125312 bytes 2023-10-11T10:45:44.289 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1600 IOs, 633043968 bytes 2023-10-11T10:45:44.499 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1700 IOs, 706796544 bytes 2023-10-11T10:45:44.651 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1800 IOs, 755282944 bytes 2023-10-11T10:45:44.845 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1900 IOs, 821952512 bytes 2023-10-11T10:45:44.956 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2000 IOs, 834160640 bytes 2023-10-11T10:45:45.069 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2100 IOs, 859409408 bytes 2023-10-11T10:45:45.201 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2200 IOs, 889461760 bytes 2023-10-11T10:45:45.361 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2300 IOs, 935751680 bytes 2023-10-11T10:45:45.464 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2400 IOs, 960171008 bytes 2023-10-11T10:45:45.547 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2500 IOs, 960787456 bytes 2023-10-11T10:45:45.686 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2600 IOs, 1009777664 bytes 2023-10-11T10:45:45.870 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2700 IOs, 1072889856 bytes 2023-10-11T10:45:45.952 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2800 IOs, 1083116544 bytes 2023-10-11T10:45:46.030 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2900 IOs, 1089652736 bytes 2023-10-11T10:45:46.168 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3000 IOs, 1144175616 bytes 2023-10-11T10:45:46.345 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3100 IOs, 1192007680 bytes 2023-10-11T10:45:46.493 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.5015990203444102) > weight (0.25) 2023-10-11T10:45:46.493 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.5152696063954656) > weight (0.25) 2023-10-11T10:45:46.493 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.2 2023-10-11T10:45:46.494 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Sent signal 15 2023-10-11T10:45:46.494 INFO:tasks.rbd_mirror_thrash:waiting for 8 secs before reviving daemons 2023-10-11T10:45:46.495 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi111.stderr:2023-10-11T10:45:46.494+0000 7f008e92a700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 (PID: 106685) UID: 1000 2023-10-11T10:45:46.509 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3200 IOs, 1239524352 bytes 2023-10-11T10:45:46.705 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3300 IOs, 1291509760 bytes 2023-10-11T10:45:46.826 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3400 IOs, 1317936128 bytes 2023-10-11T10:45:46.838 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ error_code=124 2023-10-11T10:45:46.838 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ set -e 2023-10-11T10:45:46.838 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' 124 -eq 124 ']' 2023-10-11T10:45:46.838 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T10:45:46.838 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2023-10-11T10:45:46.839 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T10:45:46.839 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:45:46.839 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:45:46.839 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local state_pattern=up+replaying 2023-10-11T10:45:46.839 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local description_pattern= 2023-10-11T10:45:46.840 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local service_pattern= 2023-10-11T10:45:46.840 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-10-11T10:45:46.840 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 1 2023-10-11T10:45:47.839 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2023-10-11T10:45:47.840 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T10:45:47.840 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:45:47.840 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:45:47.840 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local state_pattern=up+replaying 2023-10-11T10:45:47.840 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local description_pattern= 2023-10-11T10:45:47.841 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local service_pattern= 2023-10-11T10:45:47.841 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ mkfname cluster1-mirror-test.mirror_status 2023-10-11T10:45:47.841 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ echo cluster1-mirror-test.mirror_status 2023-10-11T10:45:47.841 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -e 's|[/ ]|_|g' 2023-10-11T10:45:47.842 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local status_log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.mirror_status 2023-10-11T10:45:47.842 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ CEPH_ARGS= 2023-10-11T10:45:47.843 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2023-10-11T10:45:47.843 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ tee /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.mirror_status 2023-10-11T10:45:47.891 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:test: 2023-10-11T10:45:47.892 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: global_id: 3dd93db4-c9cc-462c-8b49-007eba9a90d2 2023-10-11T10:45:47.892 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: state: up+replaying 2023-10-11T10:45:47.892 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1697021085,"remote_snapshot_timestamp":1697021085,"replay_state":"idle"} 2023-10-11T10:45:47.892 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: service: mirror.1 on smithi111 2023-10-11T10:45:47.893 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: last_update: 2023-10-11 10:45:21 2023-10-11T10:45:47.893 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: peer_sites: 2023-10-11T10:45:47.893 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: name: cluster2 2023-10-11T10:45:47.893 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: state: down+stopped 2023-10-11T10:45:47.894 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: description: local image is primary 2023-10-11T10:45:47.894 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: last_update: 2023-10-11 10:45:21 2023-10-11T10:45:47.897 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.mirror_status 2023-10-11T10:45:47.898 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: state: up+replaying 2023-10-11T10:45:47.899 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ grep '^ description: .*' /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.mirror_status 2023-10-11T10:45:47.899 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1697021085,"remote_snapshot_timestamp":1697021085,"replay_state":"idle"} 2023-10-11T10:45:47.900 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n '' ']' 2023-10-11T10:45:47.900 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ echo up+replaying 2023-10-11T10:45:47.900 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ grep '^up+' 2023-10-11T10:45:47.901 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:up+replaying 2023-10-11T10:45:47.901 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ grep 'service: *mirror..* on ' /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.mirror_status 2023-10-11T10:45:47.902 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: service: mirror.1 on smithi111 2023-10-11T10:45:47.903 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.mirror_status 2023-10-11T10:45:47.903 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local 'last_update=2023-10-11 10:45:21' 2023-10-11T10:45:47.904 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2023-10-11 10:45:21' 2023-10-11T10:45:47.904 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T10:45:47.904 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:45:47.904 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:45:47.905 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local state_pattern=up+replaying 2023-10-11T10:45:47.905 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local 'prev_last_update=2023-10-11 10:45:21' 2023-10-11T10:45:47.905 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ mkfname cluster1-mirror.mirror_status 2023-10-11T10:45:47.905 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ echo cluster1-mirror.mirror_status 2023-10-11T10:45:47.905 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -e 's|[/ ]|_|g' 2023-10-11T10:45:47.906 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local status_log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror.mirror_status 2023-10-11T10:45:47.906 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2023-10-11T10:45:47.941 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd: failed to query services: (13) Permission denied 2023-10-11T10:45:47.961 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local last_update state 2023-10-11T10:45:47.962 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2023-10-11T10:45:47.964 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ last_update='2023-10-11 10:45:21' 2023-10-11T10:45:47.964 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2023-10-11T10:45:47.966 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ state=up+replaying 2023-10-11T10:45:47.966 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ echo up+replaying 2023-10-11T10:45:47.966 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ grep up+replaying 2023-10-11T10:45:47.967 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:up+replaying 2023-10-11T10:45:47.968 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T10:45:47.968 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T10:45:47.968 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ snap_name=snap2 2023-10-11T10:45:47.968 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ create_snap cluster2 mirror test snap2 2023-10-11T10:45:47.969 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:45:47.969 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:45:47.969 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:45:47.969 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap2 2023-10-11T10:45:47.969 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap2 --debug-rbd=20 --debug-journaler=20 2023-10-11T10:45:54.495 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.2 2023-10-11T10:45:54.496 INFO:teuthology.orchestra.run:waiting for 600 2023-10-11T10:45:54.496 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.2 2023-10-11T10:45:54.496 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Restarting daemon 2023-10-11T10:45:54.497 DEBUG:teuthology.orchestra.run.smithi111:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 2023-10-11T10:45:54.500 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Started 2023-10-11T10:45:54.500 INFO:tasks.rbd_mirror_thrash:waiting for 73 secs before thrashing 2023-10-11T10:46:17.825 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_replay_started cluster1 mirror test 2023-10-11T10:46:17.825 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T10:46:17.825 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:46:17.826 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:46:17.826 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2023-10-11T10:46:17.826 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T10:46:17.826 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:46:17.826 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:46:17.827 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local state=started 2023-10-11T10:46:17.827 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T10:46:17.827 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-10-11T10:46:17.827 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 1 2023-10-11T10:46:18.826 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test_image_replay_state cluster1 mirror test started 2023-10-11T10:46:18.826 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T10:46:18.826 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:46:18.827 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:46:18.827 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local test_state=started 2023-10-11T10:46:18.827 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local status_result 2023-10-11T10:46:18.827 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_state=stopped 2023-10-11T10:46:18.830 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2023-10-11T10:46:18.831 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_instance=cluster1 2023-10-11T10:46:18.831 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ grep -i state 2023-10-11T10:46:18.831 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T10:46:18.831 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T10:46:18.831 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance=cluster1 2023-10-11T10:46:18.832 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local loop_instance 2023-10-11T10:46:18.832 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-10-11T10:46:18.832 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sleep 0 2023-10-11T10:46:18.832 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-10-11T10:46:18.832 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ seq 0 3 2023-10-11T10:46:18.833 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-10-11T10:46:18.833 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-10-11T10:46:18.833 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1:0 2023-10-11T10:46:18.833 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T10:46:18.833 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance 2023-10-11T10:46:18.834 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-10-11T10:46:18.834 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local val=cluster1:0 2023-10-11T10:46:18.834 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_var_name=cluster 2023-10-11T10:46:18.834 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance_var_name=instance 2023-10-11T10:46:18.835 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ cluster=cluster1 2023-10-11T10:46:18.835 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ instance=0 2023-10-11T10:46:18.835 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' 0 = cluster1:0 ']' 2023-10-11T10:46:18.835 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval cluster=cluster1 2023-10-11T10:46:18.835 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T10:46:18.836 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval instance=0 2023-10-11T10:46:18.836 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=0 2023-10-11T10:46:18.836 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-10-11T10:46:18.836 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local local_cluster=cluster1:0 2023-10-11T10:46:18.836 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster=cluster1 2023-10-11T10:46:18.837 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance 2023-10-11T10:46:18.837 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-10-11T10:46:18.837 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local val=cluster1:0 2023-10-11T10:46:18.837 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster_var_name=local_cluster 2023-10-11T10:46:18.838 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance_var_name=instance 2023-10-11T10:46:18.839 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T10:46:18.839 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=0 2023-10-11T10:46:18.839 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ '[' 0 = cluster1:0 ']' 2023-10-11T10:46:18.839 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval local_cluster=cluster1 2023-10-11T10:46:18.839 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ local_cluster=cluster1 2023-10-11T10:46:18.840 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval instance=0 2023-10-11T10:46:18.840 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ instance=0 2023-10-11T10:46:18.840 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-10-11T10:46:18.853 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T10:46:18.853 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T10:46:18.853 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T10:46:18.853 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-10-11T10:46:18.954 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-10-11T10:46:18.954 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd mirror status : get status for rbd mirror 2023-10-11T10:46:18.954 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:admin_socket: invalid command 2023-10-11T10:46:18.958 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-10-11T10:46:18.958 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-10-11T10:46:18.958 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1:1 2023-10-11T10:46:18.959 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T10:46:18.959 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance 2023-10-11T10:46:18.959 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-10-11T10:46:18.959 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local val=cluster1:1 2023-10-11T10:46:18.960 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_var_name=cluster 2023-10-11T10:46:18.960 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance_var_name=instance 2023-10-11T10:46:18.960 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ cluster=cluster1 2023-10-11T10:46:18.960 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ instance=1 2023-10-11T10:46:18.961 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' 1 = cluster1:1 ']' 2023-10-11T10:46:18.961 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval cluster=cluster1 2023-10-11T10:46:18.961 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T10:46:18.961 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval instance=1 2023-10-11T10:46:18.961 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=1 2023-10-11T10:46:18.962 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-10-11T10:46:18.962 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local local_cluster=cluster1:1 2023-10-11T10:46:18.962 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster=cluster1 2023-10-11T10:46:18.962 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance 2023-10-11T10:46:18.962 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-10-11T10:46:18.963 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local val=cluster1:1 2023-10-11T10:46:18.963 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster_var_name=local_cluster 2023-10-11T10:46:18.963 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance_var_name=instance 2023-10-11T10:46:18.963 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T10:46:18.963 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=1 2023-10-11T10:46:18.964 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ '[' 1 = cluster1:1 ']' 2023-10-11T10:46:18.964 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval local_cluster=cluster1 2023-10-11T10:46:18.964 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ local_cluster=cluster1 2023-10-11T10:46:18.964 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval instance=1 2023-10-11T10:46:18.964 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ instance=1 2023-10-11T10:46:18.965 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-10-11T10:46:18.976 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T10:46:18.976 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T10:46:18.976 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T10:46:18.977 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-10-11T10:46:19.079 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ return 0 2023-10-11T10:46:19.080 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ status_result=' "state": "Replaying"' 2023-10-11T10:46:19.081 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ echo ' "state": "Replaying"' 2023-10-11T10:46:19.081 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ grep -i Replaying 2023-10-11T10:46:19.081 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: "state": "Replaying" 2023-10-11T10:46:19.083 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ current_state=started 2023-10-11T10:46:19.083 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test started = started 2023-10-11T10:46:19.083 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T10:46:19.084 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2023-10-11T10:46:19.084 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local local_cluster=cluster1 2023-10-11T10:46:19.084 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:46:19.084 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:46:19.084 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:46:19.085 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' snapshot = journal ']' 2023-10-11T10:46:19.085 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' snapshot = snapshot ']' 2023-10-11T10:46:19.085 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2023-10-11T10:46:19.085 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local local_cluster=cluster1 2023-10-11T10:46:19.086 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:46:19.086 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:46:19.086 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:46:19.086 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ mkfname cluster2-mirror-test.status 2023-10-11T10:46:19.088 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ echo cluster2-mirror-test.status 2023-10-11T10:46:19.088 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -e 's|[/ ]|_|g' 2023-10-11T10:46:19.088 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local status_log=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.status 2023-10-11T10:46:19.088 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ mkfname cluster1-mirror-test.status 2023-10-11T10:46:19.089 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ echo cluster1-mirror-test.status 2023-10-11T10:46:19.089 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -e 's|[/ ]|_|g' 2023-10-11T10:46:19.089 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local local_status_log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.status 2023-10-11T10:46:19.089 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ mirror_image_snapshot cluster2 mirror test 2023-10-11T10:46:19.089 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:46:19.089 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:46:19.090 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:46:19.090 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2023-10-11T10:46:20.833 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:Snapshot ID: 9 2023-10-11T10:46:20.847 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.status 2023-10-11T10:46:20.847 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:46:20.847 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:46:20.848 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:46:20.848 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.status 2023-10-11T10:46:20.848 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2023-10-11T10:46:20.848 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T10:46:20.893 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-10-11T10:46:20.895 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snapshot_id=9 2023-10-11T10:46:20.895 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T10:46:20.895 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T10:46:20.895 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.2 2023-10-11T10:46:21.096 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.status 2023-10-11T10:46:21.096 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T10:46:21.096 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:46:21.097 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:46:21.097 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.status 2023-10-11T10:46:21.097 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-10-11T10:46:21.097 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T10:46:21.144 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T10:46:21.145 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T10:46:21.146 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:-:1.1: Document is empty 2023-10-11T10:46:21.146 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: 2023-10-11T10:46:21.147 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:^ 2023-10-11T10:46:21.147 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id= 2023-10-11T10:46:21.147 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 9 = '' 2023-10-11T10:46:21.148 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T10:46:21.148 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.4 2023-10-11T10:46:21.548 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.status 2023-10-11T10:46:21.548 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T10:46:21.548 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:46:21.549 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:46:21.549 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.status 2023-10-11T10:46:21.549 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-10-11T10:46:21.549 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T10:46:21.602 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T10:46:21.603 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T10:46:21.604 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:-:1.1: Document is empty 2023-10-11T10:46:21.604 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: 2023-10-11T10:46:21.605 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:^ 2023-10-11T10:46:21.605 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id= 2023-10-11T10:46:21.606 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 9 = '' 2023-10-11T10:46:21.606 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T10:46:21.606 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.8 2023-10-11T10:46:22.406 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.status 2023-10-11T10:46:22.407 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T10:46:22.407 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:46:22.407 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:46:22.407 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.status 2023-10-11T10:46:22.407 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-10-11T10:46:22.408 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T10:46:22.458 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T10:46:22.459 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T10:46:22.460 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:-:1.1: Document is empty 2023-10-11T10:46:22.460 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: 2023-10-11T10:46:22.460 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:^ 2023-10-11T10:46:22.461 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id= 2023-10-11T10:46:22.461 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 9 = '' 2023-10-11T10:46:22.461 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T10:46:22.461 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 1.6 2023-10-11T10:46:24.062 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.status 2023-10-11T10:46:24.062 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T10:46:24.063 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:46:24.063 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:46:24.063 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.status 2023-10-11T10:46:24.063 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-10-11T10:46:24.063 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T10:46:24.107 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T10:46:24.110 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id=9 2023-10-11T10:46:24.110 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 9 = 9 2023-10-11T10:46:24.110 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T10:46:24.110 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_snap_present cluster1 mirror test snap2 2023-10-11T10:46:24.111 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T10:46:24.111 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:46:24.111 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:46:24.111 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap2 2023-10-11T10:46:24.111 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T10:46:24.111 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-10-11T10:46:24.112 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 1 2023-10-11T10:46:25.111 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 info mirror/test@snap2 2023-10-11T10:46:25.157 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:rbd image 'test': 2023-10-11T10:46:25.157 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: size 512 MiB in 128 objects 2023-10-11T10:46:25.158 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: order 22 (4 MiB objects) 2023-10-11T10:46:25.158 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: snapshot_count: 4 2023-10-11T10:46:25.158 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: id: 10afefb6b972 2023-10-11T10:46:25.158 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: block_name_prefix: rbd_data.10afefb6b972 2023-10-11T10:46:25.159 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: format: 2 2023-10-11T10:46:25.159 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-10-11T10:46:25.159 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: op_features: 2023-10-11T10:46:25.159 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: flags: 2023-10-11T10:46:25.159 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: create_timestamp: Wed Oct 11 10:44:53 2023 2023-10-11T10:46:25.160 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: access_timestamp: Wed Oct 11 10:44:53 2023 2023-10-11T10:46:25.160 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: modify_timestamp: Wed Oct 11 10:44:53 2023 2023-10-11T10:46:25.160 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: protected: False 2023-10-11T10:46:25.160 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring state: enabled 2023-10-11T10:46:25.161 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring mode: snapshot 2023-10-11T10:46:25.161 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring global id: 3dd93db4-c9cc-462c-8b49-007eba9a90d2 2023-10-11T10:46:25.161 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring primary: false 2023-10-11T10:46:25.162 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T10:46:25.162 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n snap1-clean ']' 2023-10-11T10:46:25.163 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ compare_image_snaps mirror test snap1-clean 2023-10-11T10:46:25.163 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:46:25.163 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:46:25.163 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap1-clean 2023-10-11T10:46:25.163 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local ret=0 2023-10-11T10:46:25.163 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local rmt_export=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export 2023-10-11T10:46:25.164 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local loc_export=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:46:25.164 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rm -f /tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:46:25.164 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 -p mirror export test@snap1-clean /tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export 2023-10-11T10:46:25.721 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-10-11T10:46:25.728 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 -p mirror export test@snap1-clean /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:46:26.454 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-10-11T10:46:26.464 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ cmp /tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:46:26.682 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rm -f /tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:46:26.821 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T10:46:26.822 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ compare_image_snaps mirror test snap2 2023-10-11T10:46:26.822 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:46:26.822 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:46:26.822 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap2 2023-10-11T10:46:26.822 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local ret=0 2023-10-11T10:46:26.823 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local rmt_export=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export 2023-10-11T10:46:26.823 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local loc_export=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:46:26.823 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rm -f /tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:46:26.823 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 -p mirror export test@snap2 /tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export 2023-10-11T10:46:27.394 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-10-11T10:46:27.402 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 -p mirror export test@snap2 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:46:28.351 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-10-11T10:46:28.360 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ cmp /tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:46:28.572 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rm -f /tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:46:28.713 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T10:46:28.713 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ clean_snap_name=snap2-clean 2023-10-11T10:46:28.713 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ create_snap cluster2 mirror test snap2-clean 2023-10-11T10:46:28.713 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:46:28.714 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:46:28.714 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:46:28.714 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap2-clean 2023-10-11T10:46:28.714 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap2-clean --debug-rbd=20 --debug-journaler=20 2023-10-11T10:46:28.848 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 10` 2023-10-11T10:46:28.848 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ stress_write_image cluster2 mirror test 2023-10-11T10:46:28.849 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:46:28.849 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:46:28.849 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:46:28.849 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-10-11T10:46:28.850 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local duration=13 2023-10-11T10:46:28.851 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ set +e 2023-10-11T10:46:28.851 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2023-10-11T10:46:29.165 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:100 IOs, 65718272 bytes 2023-10-11T10:46:29.276 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:200 IOs, 87753728 bytes 2023-10-11T10:46:29.430 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:300 IOs, 130852864 bytes 2023-10-11T10:46:29.605 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:400 IOs, 172723200 bytes 2023-10-11T10:46:29.681 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:500 IOs, 177952768 bytes 2023-10-11T10:46:29.791 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:600 IOs, 199697408 bytes 2023-10-11T10:46:30.074 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:700 IOs, 316064768 bytes 2023-10-11T10:46:30.178 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:800 IOs, 335866880 bytes 2023-10-11T10:46:30.311 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:900 IOs, 370607104 bytes 2023-10-11T10:46:30.418 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1000 IOs, 392114176 bytes 2023-10-11T10:46:30.522 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1100 IOs, 414355456 bytes 2023-10-11T10:46:30.631 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1200 IOs, 438269952 bytes 2023-10-11T10:46:30.736 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1300 IOs, 454712320 bytes 2023-10-11T10:46:30.851 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1400 IOs, 480558080 bytes 2023-10-11T10:46:31.023 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1500 IOs, 527960064 bytes 2023-10-11T10:46:31.096 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1600 IOs, 528580608 bytes 2023-10-11T10:46:31.331 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1700 IOs, 617390080 bytes 2023-10-11T10:46:31.503 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1800 IOs, 672779264 bytes 2023-10-11T10:46:31.580 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1900 IOs, 687576064 bytes 2023-10-11T10:46:31.780 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2000 IOs, 739500032 bytes 2023-10-11T10:46:31.963 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2100 IOs, 805165056 bytes 2023-10-11T10:46:32.164 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2200 IOs, 891859968 bytes 2023-10-11T10:46:32.338 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2300 IOs, 936044544 bytes 2023-10-11T10:46:32.435 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2400 IOs, 956045312 bytes 2023-10-11T10:46:32.548 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2500 IOs, 974599168 bytes 2023-10-11T10:46:32.724 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2600 IOs, 1051958272 bytes 2023-10-11T10:46:32.885 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2700 IOs, 1083657216 bytes 2023-10-11T10:46:33.056 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2800 IOs, 1132624896 bytes 2023-10-11T10:46:33.207 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2900 IOs, 1177674752 bytes 2023-10-11T10:46:33.444 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3000 IOs, 1256171520 bytes 2023-10-11T10:46:33.637 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3100 IOs, 1320719360 bytes 2023-10-11T10:46:33.837 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3200 IOs, 1409707008 bytes 2023-10-11T10:46:34.029 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3300 IOs, 1463857152 bytes 2023-10-11T10:46:34.149 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3400 IOs, 1489167360 bytes 2023-10-11T10:46:34.320 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3500 IOs, 1538932736 bytes 2023-10-11T10:46:34.418 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3600 IOs, 1556204544 bytes 2023-10-11T10:46:34.560 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3700 IOs, 1598481408 bytes 2023-10-11T10:46:34.662 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3800 IOs, 1609390080 bytes 2023-10-11T10:46:34.880 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3900 IOs, 1689447424 bytes 2023-10-11T10:46:34.947 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4000 IOs, 1690048512 bytes 2023-10-11T10:46:35.065 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4100 IOs, 1711488000 bytes 2023-10-11T10:46:35.166 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4200 IOs, 1725596672 bytes 2023-10-11T10:46:35.258 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4300 IOs, 1758878720 bytes 2023-10-11T10:46:35.416 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4400 IOs, 1795929088 bytes 2023-10-11T10:46:35.489 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4500 IOs, 1812756480 bytes 2023-10-11T10:46:35.608 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4600 IOs, 1833474048 bytes 2023-10-11T10:46:35.728 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4700 IOs, 1864704000 bytes 2023-10-11T10:46:35.827 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4800 IOs, 1886241792 bytes 2023-10-11T10:46:35.966 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4900 IOs, 1924929536 bytes 2023-10-11T10:46:36.143 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5000 IOs, 1984640000 bytes 2023-10-11T10:46:36.286 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5100 IOs, 2024142848 bytes 2023-10-11T10:46:36.397 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5200 IOs, 2072113152 bytes 2023-10-11T10:46:36.570 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5300 IOs, 2099999744 bytes 2023-10-11T10:46:36.727 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5400 IOs, 2147543040 bytes 2023-10-11T10:46:36.874 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5500 IOs, 2194136064 bytes 2023-10-11T10:46:37.026 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5600 IOs, 2242625536 bytes 2023-10-11T10:46:37.113 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5700 IOs, 2269686784 bytes 2023-10-11T10:46:37.373 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5800 IOs, 2357123072 bytes 2023-10-11T10:46:37.481 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5900 IOs, 2388645888 bytes 2023-10-11T10:46:37.637 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6000 IOs, 2434711552 bytes 2023-10-11T10:46:37.818 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6100 IOs, 2495388672 bytes 2023-10-11T10:46:37.998 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6200 IOs, 2545164288 bytes 2023-10-11T10:46:38.113 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6300 IOs, 2579296256 bytes 2023-10-11T10:46:38.237 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6400 IOs, 2596859904 bytes 2023-10-11T10:46:38.418 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6500 IOs, 2642386944 bytes 2023-10-11T10:46:38.575 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6600 IOs, 2686886912 bytes 2023-10-11T10:46:38.718 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6700 IOs, 2721895424 bytes 2023-10-11T10:46:39.013 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6800 IOs, 2836367360 bytes 2023-10-11T10:46:39.160 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6900 IOs, 2870227968 bytes 2023-10-11T10:46:39.349 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:7000 IOs, 2940003328 bytes 2023-10-11T10:46:39.439 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:7100 IOs, 2967405568 bytes 2023-10-11T10:46:39.495 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.3556896717155439) > weight (0.25) 2023-10-11T10:46:39.496 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.7530077585996607) > weight (0.25) 2023-10-11T10:46:39.496 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.6434769886839056) > weight (0.25) 2023-10-11T10:46:39.496 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.7762345927180944) > weight (0.25) 2023-10-11T10:46:39.496 INFO:tasks.rbd_mirror_thrash:waiting for 79 secs before thrashing 2023-10-11T10:46:39.717 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:7200 IOs, 3053093888 bytes 2023-10-11T10:46:39.885 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:7300 IOs, 3098948608 bytes 2023-10-11T10:46:40.084 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:7400 IOs, 3165958144 bytes 2023-10-11T10:46:40.195 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:7500 IOs, 3191390208 bytes 2023-10-11T10:46:40.341 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:7600 IOs, 3231720448 bytes 2023-10-11T10:46:40.414 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:7700 IOs, 3232348160 bytes 2023-10-11T10:46:40.501 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:7800 IOs, 3247874048 bytes 2023-10-11T10:46:40.670 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:7900 IOs, 3302115328 bytes 2023-10-11T10:46:40.844 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:8000 IOs, 3360478208 bytes 2023-10-11T10:46:41.052 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:8100 IOs, 3430571008 bytes 2023-10-11T10:46:41.196 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:8200 IOs, 3476722688 bytes 2023-10-11T10:46:41.268 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:8300 IOs, 3483204608 bytes 2023-10-11T10:46:41.379 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:8400 IOs, 3506414592 bytes 2023-10-11T10:46:41.527 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:8500 IOs, 3552297984 bytes 2023-10-11T10:46:41.729 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:8600 IOs, 3604321280 bytes 2023-10-11T10:46:41.854 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ error_code=124 2023-10-11T10:46:41.855 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ set -e 2023-10-11T10:46:41.855 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' 124 -eq 124 ']' 2023-10-11T10:46:41.855 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T10:46:41.855 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2023-10-11T10:46:41.855 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T10:46:41.856 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:46:41.856 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:46:41.856 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local state_pattern=up+replaying 2023-10-11T10:46:41.856 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local description_pattern= 2023-10-11T10:46:41.856 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local service_pattern= 2023-10-11T10:46:41.857 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-10-11T10:46:41.857 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 1 2023-10-11T10:46:42.856 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2023-10-11T10:46:42.856 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T10:46:42.856 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:46:42.857 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:46:42.857 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local state_pattern=up+replaying 2023-10-11T10:46:42.857 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local description_pattern= 2023-10-11T10:46:42.857 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local service_pattern= 2023-10-11T10:46:42.857 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ mkfname cluster1-mirror-test.mirror_status 2023-10-11T10:46:42.858 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ echo cluster1-mirror-test.mirror_status 2023-10-11T10:46:42.858 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -e 's|[/ ]|_|g' 2023-10-11T10:46:42.858 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local status_log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.mirror_status 2023-10-11T10:46:42.859 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ CEPH_ARGS= 2023-10-11T10:46:42.859 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2023-10-11T10:46:42.859 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ tee /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.mirror_status 2023-10-11T10:46:42.908 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:test: 2023-10-11T10:46:42.909 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: global_id: 3dd93db4-c9cc-462c-8b49-007eba9a90d2 2023-10-11T10:46:42.909 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: state: up+replaying 2023-10-11T10:46:42.909 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":255178240.0,"last_snapshot_bytes":510356480,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1697021134,"remote_snapshot_timestamp":1697021134,"replay_state":"idle"} 2023-10-11T10:46:42.909 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: service: mirror.1 on smithi111 2023-10-11T10:46:42.909 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: last_update: 2023-10-11 10:46:21 2023-10-11T10:46:42.910 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: peer_sites: 2023-10-11T10:46:42.910 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: name: cluster2 2023-10-11T10:46:42.910 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: state: up+stopped 2023-10-11T10:46:42.910 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: description: local image is primary 2023-10-11T10:46:42.910 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: last_update: 2023-10-11 10:46:21 2023-10-11T10:46:42.928 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.mirror_status 2023-10-11T10:46:42.929 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: state: up+replaying 2023-10-11T10:46:42.930 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ grep '^ description: .*' /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.mirror_status 2023-10-11T10:46:42.930 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":255178240.0,"last_snapshot_bytes":510356480,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1697021134,"remote_snapshot_timestamp":1697021134,"replay_state":"idle"} 2023-10-11T10:46:42.931 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n '' ']' 2023-10-11T10:46:42.931 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ echo up+replaying 2023-10-11T10:46:42.931 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ grep '^up+' 2023-10-11T10:46:42.932 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:up+replaying 2023-10-11T10:46:42.932 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ grep 'service: *mirror..* on ' /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.mirror_status 2023-10-11T10:46:42.933 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: service: mirror.1 on smithi111 2023-10-11T10:46:42.933 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.mirror_status 2023-10-11T10:46:42.935 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local 'last_update=2023-10-11 10:46:21' 2023-10-11T10:46:42.935 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2023-10-11 10:46:21' 2023-10-11T10:46:42.935 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T10:46:42.936 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:46:42.936 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:46:42.936 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local state_pattern=up+replaying 2023-10-11T10:46:42.936 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local 'prev_last_update=2023-10-11 10:46:21' 2023-10-11T10:46:42.936 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ mkfname cluster1-mirror.mirror_status 2023-10-11T10:46:42.937 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ echo cluster1-mirror.mirror_status 2023-10-11T10:46:42.937 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -e 's|[/ ]|_|g' 2023-10-11T10:46:42.937 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local status_log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror.mirror_status 2023-10-11T10:46:42.937 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2023-10-11T10:46:42.974 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd: failed to query services: (13) Permission denied 2023-10-11T10:46:43.037 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local last_update state 2023-10-11T10:46:43.038 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2023-10-11T10:46:43.039 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ last_update='2023-10-11 10:46:21' 2023-10-11T10:46:43.040 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2023-10-11T10:46:43.041 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ state=up+replaying 2023-10-11T10:46:43.042 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ echo up+replaying 2023-10-11T10:46:43.042 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ grep up+replaying 2023-10-11T10:46:43.042 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:up+replaying 2023-10-11T10:46:43.043 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T10:46:43.043 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T10:46:43.044 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ snap_name=snap3 2023-10-11T10:46:43.044 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ create_snap cluster2 mirror test snap3 2023-10-11T10:46:43.044 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:46:43.044 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:46:43.044 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:46:43.045 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap3 2023-10-11T10:46:43.045 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap3 --debug-rbd=20 --debug-journaler=20 2023-10-11T10:47:07.502 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.9210319963770456) > weight (0.25) 2023-10-11T10:47:07.502 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.6425831625653534) > weight (0.25) 2023-10-11T10:47:07.503 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.9871574776041837) > weight (0.25) 2023-10-11T10:47:07.503 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.5713038160514556) > weight (0.25) 2023-10-11T10:47:07.504 INFO:tasks.rbd_mirror_thrash:waiting for 96 secs before thrashing 2023-10-11T10:47:13.837 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_replay_started cluster1 mirror test 2023-10-11T10:47:13.838 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T10:47:13.838 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:47:13.838 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:47:13.838 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2023-10-11T10:47:13.839 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T10:47:13.839 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:47:13.839 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:47:13.839 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local state=started 2023-10-11T10:47:13.839 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T10:47:13.840 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-10-11T10:47:13.840 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 1 2023-10-11T10:47:14.839 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test_image_replay_state cluster1 mirror test started 2023-10-11T10:47:14.839 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T10:47:14.840 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:47:14.840 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:47:14.840 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local test_state=started 2023-10-11T10:47:14.840 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local status_result 2023-10-11T10:47:14.840 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_state=stopped 2023-10-11T10:47:14.840 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2023-10-11T10:47:14.841 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_instance=cluster1 2023-10-11T10:47:14.841 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ grep -i state 2023-10-11T10:47:14.841 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T10:47:14.841 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T10:47:14.841 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance=cluster1 2023-10-11T10:47:14.842 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local loop_instance 2023-10-11T10:47:14.842 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-10-11T10:47:14.842 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sleep 0 2023-10-11T10:47:14.844 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-10-11T10:47:14.844 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ seq 0 3 2023-10-11T10:47:14.844 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-10-11T10:47:14.844 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-10-11T10:47:14.844 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1:0 2023-10-11T10:47:14.845 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T10:47:14.845 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance 2023-10-11T10:47:14.845 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-10-11T10:47:14.845 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local val=cluster1:0 2023-10-11T10:47:14.845 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_var_name=cluster 2023-10-11T10:47:14.845 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance_var_name=instance 2023-10-11T10:47:14.846 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ cluster=cluster1 2023-10-11T10:47:14.846 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ instance=0 2023-10-11T10:47:14.846 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' 0 = cluster1:0 ']' 2023-10-11T10:47:14.846 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval cluster=cluster1 2023-10-11T10:47:14.846 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T10:47:14.846 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval instance=0 2023-10-11T10:47:14.847 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=0 2023-10-11T10:47:14.847 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-10-11T10:47:14.847 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local local_cluster=cluster1:0 2023-10-11T10:47:14.847 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster=cluster1 2023-10-11T10:47:14.847 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance 2023-10-11T10:47:14.849 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-10-11T10:47:14.849 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local val=cluster1:0 2023-10-11T10:47:14.849 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster_var_name=local_cluster 2023-10-11T10:47:14.849 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance_var_name=instance 2023-10-11T10:47:14.849 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T10:47:14.850 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=0 2023-10-11T10:47:14.850 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ '[' 0 = cluster1:0 ']' 2023-10-11T10:47:14.850 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval local_cluster=cluster1 2023-10-11T10:47:14.850 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ local_cluster=cluster1 2023-10-11T10:47:14.850 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval instance=0 2023-10-11T10:47:14.851 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ instance=0 2023-10-11T10:47:14.851 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-10-11T10:47:14.865 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T10:47:14.865 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T10:47:14.865 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T10:47:14.866 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-10-11T10:47:14.966 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-10-11T10:47:14.967 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd mirror status : get status for rbd mirror 2023-10-11T10:47:14.967 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:admin_socket: invalid command 2023-10-11T10:47:14.970 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-10-11T10:47:14.970 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-10-11T10:47:14.970 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1:1 2023-10-11T10:47:14.970 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T10:47:14.971 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance 2023-10-11T10:47:14.971 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-10-11T10:47:14.971 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local val=cluster1:1 2023-10-11T10:47:14.971 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_var_name=cluster 2023-10-11T10:47:14.971 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance_var_name=instance 2023-10-11T10:47:14.972 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ cluster=cluster1 2023-10-11T10:47:14.972 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ instance=1 2023-10-11T10:47:14.972 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' 1 = cluster1:1 ']' 2023-10-11T10:47:14.972 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval cluster=cluster1 2023-10-11T10:47:14.972 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T10:47:14.972 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval instance=1 2023-10-11T10:47:14.973 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=1 2023-10-11T10:47:14.973 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-10-11T10:47:14.973 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local local_cluster=cluster1:1 2023-10-11T10:47:14.973 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster=cluster1 2023-10-11T10:47:14.973 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance 2023-10-11T10:47:14.973 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-10-11T10:47:14.974 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local val=cluster1:1 2023-10-11T10:47:14.974 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster_var_name=local_cluster 2023-10-11T10:47:14.974 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance_var_name=instance 2023-10-11T10:47:14.974 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T10:47:14.974 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=1 2023-10-11T10:47:14.975 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ '[' 1 = cluster1:1 ']' 2023-10-11T10:47:14.975 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval local_cluster=cluster1 2023-10-11T10:47:14.975 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ local_cluster=cluster1 2023-10-11T10:47:14.975 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval instance=1 2023-10-11T10:47:14.975 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ instance=1 2023-10-11T10:47:14.975 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-10-11T10:47:14.989 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T10:47:14.989 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T10:47:14.989 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T10:47:14.989 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-10-11T10:47:15.092 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ return 0 2023-10-11T10:47:15.093 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ status_result=' "state": "Replaying"' 2023-10-11T10:47:15.093 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ echo ' "state": "Replaying"' 2023-10-11T10:47:15.093 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ grep -i Replaying 2023-10-11T10:47:15.094 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: "state": "Replaying" 2023-10-11T10:47:15.095 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ current_state=started 2023-10-11T10:47:15.095 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test started = started 2023-10-11T10:47:15.095 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T10:47:15.096 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2023-10-11T10:47:15.096 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local local_cluster=cluster1 2023-10-11T10:47:15.096 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:47:15.096 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:47:15.096 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:47:15.097 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' snapshot = journal ']' 2023-10-11T10:47:15.097 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' snapshot = snapshot ']' 2023-10-11T10:47:15.097 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2023-10-11T10:47:15.097 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local local_cluster=cluster1 2023-10-11T10:47:15.097 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:47:15.097 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:47:15.098 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:47:15.098 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ mkfname cluster2-mirror-test.status 2023-10-11T10:47:15.098 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ echo cluster2-mirror-test.status 2023-10-11T10:47:15.098 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -e 's|[/ ]|_|g' 2023-10-11T10:47:15.099 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local status_log=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.status 2023-10-11T10:47:15.100 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ mkfname cluster1-mirror-test.status 2023-10-11T10:47:15.100 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ echo cluster1-mirror-test.status 2023-10-11T10:47:15.100 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -e 's|[/ ]|_|g' 2023-10-11T10:47:15.100 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local local_status_log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.status 2023-10-11T10:47:15.100 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ mirror_image_snapshot cluster2 mirror test 2023-10-11T10:47:15.100 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:47:15.101 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:47:15.101 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:47:15.101 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2023-10-11T10:47:16.834 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:Snapshot ID: 13 2023-10-11T10:47:16.850 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.status 2023-10-11T10:47:16.850 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:47:16.850 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:47:16.850 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:47:16.850 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.status 2023-10-11T10:47:16.851 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2023-10-11T10:47:16.851 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T10:47:16.922 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-10-11T10:47:16.924 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snapshot_id=13 2023-10-11T10:47:16.924 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T10:47:16.924 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T10:47:16.924 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.2 2023-10-11T10:47:17.125 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.status 2023-10-11T10:47:17.125 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T10:47:17.125 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:47:17.125 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:47:17.126 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.status 2023-10-11T10:47:17.126 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-10-11T10:47:17.126 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T10:47:17.172 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T10:47:17.172 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T10:47:17.173 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:-:1.1: Document is empty 2023-10-11T10:47:17.174 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: 2023-10-11T10:47:17.174 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:^ 2023-10-11T10:47:17.174 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id= 2023-10-11T10:47:17.175 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 13 = '' 2023-10-11T10:47:17.175 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T10:47:17.175 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.4 2023-10-11T10:47:17.575 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.status 2023-10-11T10:47:17.576 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T10:47:17.576 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:47:17.576 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:47:17.576 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.status 2023-10-11T10:47:17.576 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-10-11T10:47:17.576 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T10:47:17.620 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T10:47:17.621 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T10:47:17.622 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:-:1.1: Document is empty 2023-10-11T10:47:17.622 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: 2023-10-11T10:47:17.622 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:^ 2023-10-11T10:47:17.623 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id= 2023-10-11T10:47:17.623 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 13 = '' 2023-10-11T10:47:17.623 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T10:47:17.623 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.8 2023-10-11T10:47:18.424 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.status 2023-10-11T10:47:18.424 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T10:47:18.424 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:47:18.424 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:47:18.424 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.status 2023-10-11T10:47:18.425 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-10-11T10:47:18.425 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T10:47:18.473 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T10:47:18.474 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T10:47:18.475 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:-:1.1: Document is empty 2023-10-11T10:47:18.475 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: 2023-10-11T10:47:18.476 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:^ 2023-10-11T10:47:18.476 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id= 2023-10-11T10:47:18.476 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 13 = '' 2023-10-11T10:47:18.476 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T10:47:18.476 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 1.6 2023-10-11T10:47:20.077 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.status 2023-10-11T10:47:20.077 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T10:47:20.078 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:47:20.078 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:47:20.078 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.status 2023-10-11T10:47:20.078 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-10-11T10:47:20.078 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T10:47:20.122 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T10:47:20.124 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id=13 2023-10-11T10:47:20.125 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 13 = 13 2023-10-11T10:47:20.125 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T10:47:20.125 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_snap_present cluster1 mirror test snap3 2023-10-11T10:47:20.125 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T10:47:20.125 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:47:20.126 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:47:20.126 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap3 2023-10-11T10:47:20.126 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T10:47:20.126 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-10-11T10:47:20.126 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 1 2023-10-11T10:47:21.125 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 info mirror/test@snap3 2023-10-11T10:47:21.173 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:rbd image 'test': 2023-10-11T10:47:21.174 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: size 512 MiB in 128 objects 2023-10-11T10:47:21.174 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: order 22 (4 MiB objects) 2023-10-11T10:47:21.174 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: snapshot_count: 6 2023-10-11T10:47:21.174 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: id: 10afefb6b972 2023-10-11T10:47:21.174 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: block_name_prefix: rbd_data.10afefb6b972 2023-10-11T10:47:21.175 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: format: 2 2023-10-11T10:47:21.175 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-10-11T10:47:21.175 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: op_features: 2023-10-11T10:47:21.175 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: flags: 2023-10-11T10:47:21.175 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: create_timestamp: Wed Oct 11 10:44:53 2023 2023-10-11T10:47:21.175 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: access_timestamp: Wed Oct 11 10:46:25 2023 2023-10-11T10:47:21.176 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: modify_timestamp: Wed Oct 11 10:44:53 2023 2023-10-11T10:47:21.176 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: protected: False 2023-10-11T10:47:21.176 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring state: enabled 2023-10-11T10:47:21.176 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring mode: snapshot 2023-10-11T10:47:21.176 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring global id: 3dd93db4-c9cc-462c-8b49-007eba9a90d2 2023-10-11T10:47:21.177 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring primary: false 2023-10-11T10:47:21.178 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T10:47:21.178 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n snap2-clean ']' 2023-10-11T10:47:21.178 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ compare_image_snaps mirror test snap2-clean 2023-10-11T10:47:21.178 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:47:21.178 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:47:21.179 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap2-clean 2023-10-11T10:47:21.179 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local ret=0 2023-10-11T10:47:21.179 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local rmt_export=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export 2023-10-11T10:47:21.179 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local loc_export=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:47:21.179 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rm -f /tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:47:21.180 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 -p mirror export test@snap2-clean /tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export 2023-10-11T10:47:21.775 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-10-11T10:47:21.786 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 -p mirror export test@snap2-clean /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:47:22.511 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-10-11T10:47:22.518 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ cmp /tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:47:22.730 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rm -f /tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:47:22.870 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T10:47:22.870 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ compare_image_snaps mirror test snap3 2023-10-11T10:47:22.871 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:47:22.871 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:47:22.871 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap3 2023-10-11T10:47:22.871 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local ret=0 2023-10-11T10:47:22.871 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local rmt_export=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export 2023-10-11T10:47:22.872 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local loc_export=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:47:22.872 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rm -f /tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:47:22.872 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 -p mirror export test@snap3 /tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export 2023-10-11T10:47:23.434 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-10-11T10:47:23.443 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 -p mirror export test@snap3 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:47:24.117 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-10-11T10:47:24.127 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ cmp /tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:47:24.337 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rm -f /tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:47:24.478 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T10:47:24.478 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ clean_snap_name=snap3-clean 2023-10-11T10:47:24.478 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ create_snap cluster2 mirror test snap3-clean 2023-10-11T10:47:24.478 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:47:24.479 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:47:24.479 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:47:24.479 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap3-clean 2023-10-11T10:47:24.479 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap3-clean --debug-rbd=20 --debug-journaler=20 2023-10-11T10:47:24.833 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 10` 2023-10-11T10:47:24.834 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ stress_write_image cluster2 mirror test 2023-10-11T10:47:24.834 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:47:24.834 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:47:24.834 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:47:24.835 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-10-11T10:47:24.835 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local duration=5 2023-10-11T10:47:24.835 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ set +e 2023-10-11T10:47:24.836 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2023-10-11T10:47:25.113 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:100 IOs, 46781440 bytes 2023-10-11T10:47:25.241 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:200 IOs, 58430464 bytes 2023-10-11T10:47:25.351 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:300 IOs, 66144256 bytes 2023-10-11T10:47:25.478 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:400 IOs, 94327808 bytes 2023-10-11T10:47:25.585 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:500 IOs, 117256192 bytes 2023-10-11T10:47:25.694 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:600 IOs, 157350912 bytes 2023-10-11T10:47:25.974 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:700 IOs, 245841920 bytes 2023-10-11T10:47:26.151 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:800 IOs, 295745536 bytes 2023-10-11T10:47:26.306 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:900 IOs, 339753984 bytes 2023-10-11T10:47:26.474 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1000 IOs, 384891904 bytes 2023-10-11T10:47:26.630 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1100 IOs, 427867136 bytes 2023-10-11T10:47:26.764 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1200 IOs, 459067392 bytes 2023-10-11T10:47:26.876 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1300 IOs, 488520704 bytes 2023-10-11T10:47:27.031 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1400 IOs, 518903808 bytes 2023-10-11T10:47:27.130 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1500 IOs, 537321472 bytes 2023-10-11T10:47:27.204 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1600 IOs, 537907200 bytes 2023-10-11T10:47:27.308 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1700 IOs, 558112768 bytes 2023-10-11T10:47:27.524 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1800 IOs, 629000192 bytes 2023-10-11T10:47:27.658 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1900 IOs, 668164096 bytes 2023-10-11T10:47:27.765 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2000 IOs, 671098880 bytes 2023-10-11T10:47:27.893 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2100 IOs, 698953728 bytes 2023-10-11T10:47:28.075 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2200 IOs, 755780608 bytes 2023-10-11T10:47:28.265 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2300 IOs, 829174784 bytes 2023-10-11T10:47:28.403 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2400 IOs, 864997376 bytes 2023-10-11T10:47:28.509 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2500 IOs, 881318912 bytes 2023-10-11T10:47:28.700 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2600 IOs, 953385984 bytes 2023-10-11T10:47:28.775 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2700 IOs, 954001408 bytes 2023-10-11T10:47:28.976 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2800 IOs, 1021229056 bytes 2023-10-11T10:47:29.094 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2900 IOs, 1069484032 bytes 2023-10-11T10:47:29.272 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3000 IOs, 1094388736 bytes 2023-10-11T10:47:29.483 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3100 IOs, 1167963136 bytes 2023-10-11T10:47:29.753 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3200 IOs, 1259788288 bytes 2023-10-11T10:47:29.839 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ error_code=124 2023-10-11T10:47:29.840 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ set -e 2023-10-11T10:47:29.840 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' 124 -eq 124 ']' 2023-10-11T10:47:29.840 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T10:47:29.840 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2023-10-11T10:47:29.840 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T10:47:29.841 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:47:29.841 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:47:29.841 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local state_pattern=up+replaying 2023-10-11T10:47:29.841 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local description_pattern= 2023-10-11T10:47:29.841 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local service_pattern= 2023-10-11T10:47:29.842 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-10-11T10:47:29.842 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 1 2023-10-11T10:47:30.841 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2023-10-11T10:47:30.841 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T10:47:30.841 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:47:30.841 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:47:30.842 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local state_pattern=up+replaying 2023-10-11T10:47:30.842 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local description_pattern= 2023-10-11T10:47:30.842 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local service_pattern= 2023-10-11T10:47:30.842 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ mkfname cluster1-mirror-test.mirror_status 2023-10-11T10:47:30.842 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ echo cluster1-mirror-test.mirror_status 2023-10-11T10:47:30.843 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -e 's|[/ ]|_|g' 2023-10-11T10:47:30.843 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local status_log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.mirror_status 2023-10-11T10:47:30.843 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ CEPH_ARGS= 2023-10-11T10:47:30.843 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2023-10-11T10:47:30.844 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ tee /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.mirror_status 2023-10-11T10:47:30.893 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:test: 2023-10-11T10:47:30.894 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: global_id: 3dd93db4-c9cc-462c-8b49-007eba9a90d2 2023-10-11T10:47:30.894 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: state: up+replaying 2023-10-11T10:47:30.894 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: description: replaying, {"bytes_per_second":6850696.53,"bytes_per_snapshot":513039872.0,"last_snapshot_bytes":530611200,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1697021235,"remote_snapshot_timestamp":1697021235,"replay_state":"idle"} 2023-10-11T10:47:30.894 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: service: mirror.1 on smithi111 2023-10-11T10:47:30.894 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: last_update: 2023-10-11 10:47:21 2023-10-11T10:47:30.895 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: peer_sites: 2023-10-11T10:47:30.895 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: name: cluster2 2023-10-11T10:47:30.895 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: state: up+stopped 2023-10-11T10:47:30.895 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: description: local image is primary 2023-10-11T10:47:30.895 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: last_update: 2023-10-11 10:47:21 2023-10-11T10:47:30.899 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.mirror_status 2023-10-11T10:47:30.900 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: state: up+replaying 2023-10-11T10:47:30.900 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ grep '^ description: .*' /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.mirror_status 2023-10-11T10:47:30.901 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: description: replaying, {"bytes_per_second":6850696.53,"bytes_per_snapshot":513039872.0,"last_snapshot_bytes":530611200,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1697021235,"remote_snapshot_timestamp":1697021235,"replay_state":"idle"} 2023-10-11T10:47:30.901 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n '' ']' 2023-10-11T10:47:30.901 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ echo up+replaying 2023-10-11T10:47:30.902 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ grep '^up+' 2023-10-11T10:47:30.902 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:up+replaying 2023-10-11T10:47:30.902 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ grep 'service: *mirror..* on ' /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.mirror_status 2023-10-11T10:47:30.903 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: service: mirror.1 on smithi111 2023-10-11T10:47:30.903 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.mirror_status 2023-10-11T10:47:30.905 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local 'last_update=2023-10-11 10:47:21' 2023-10-11T10:47:30.905 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2023-10-11 10:47:21' 2023-10-11T10:47:30.905 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T10:47:30.906 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:47:30.906 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:47:30.906 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local state_pattern=up+replaying 2023-10-11T10:47:30.906 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local 'prev_last_update=2023-10-11 10:47:21' 2023-10-11T10:47:30.906 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ mkfname cluster1-mirror.mirror_status 2023-10-11T10:47:30.906 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ echo cluster1-mirror.mirror_status 2023-10-11T10:47:30.907 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -e 's|[/ ]|_|g' 2023-10-11T10:47:30.907 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local status_log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror.mirror_status 2023-10-11T10:47:30.907 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2023-10-11T10:47:30.943 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd: failed to query services: (13) Permission denied 2023-10-11T10:47:30.965 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local last_update state 2023-10-11T10:47:30.965 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2023-10-11T10:47:30.967 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ last_update='2023-10-11 10:47:21' 2023-10-11T10:47:30.967 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2023-10-11T10:47:30.969 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ state=up+replaying 2023-10-11T10:47:30.969 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ echo up+replaying 2023-10-11T10:47:30.970 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ grep up+replaying 2023-10-11T10:47:30.970 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:up+replaying 2023-10-11T10:47:30.971 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T10:47:30.971 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T10:47:30.971 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ snap_name=snap4 2023-10-11T10:47:30.971 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ create_snap cluster2 mirror test snap4 2023-10-11T10:47:30.972 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:47:30.972 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:47:30.972 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:47:30.972 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap4 2023-10-11T10:47:30.972 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap4 --debug-rbd=20 --debug-journaler=20 2023-10-11T10:47:56.862 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_replay_started cluster1 mirror test 2023-10-11T10:47:56.863 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T10:47:56.863 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:47:56.863 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:47:56.863 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2023-10-11T10:47:56.863 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T10:47:56.863 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:47:56.864 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:47:56.864 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local state=started 2023-10-11T10:47:56.864 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T10:47:56.864 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-10-11T10:47:56.864 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 1 2023-10-11T10:47:57.863 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test_image_replay_state cluster1 mirror test started 2023-10-11T10:47:57.863 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T10:47:57.863 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:47:57.864 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:47:57.864 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local test_state=started 2023-10-11T10:47:57.864 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local status_result 2023-10-11T10:47:57.864 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_state=stopped 2023-10-11T10:47:57.867 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2023-10-11T10:47:57.867 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ grep -i state 2023-10-11T10:47:57.867 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_instance=cluster1 2023-10-11T10:47:57.867 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T10:47:57.867 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T10:47:57.868 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance=cluster1 2023-10-11T10:47:57.868 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local loop_instance 2023-10-11T10:47:57.868 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-10-11T10:47:57.868 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sleep 0 2023-10-11T10:47:57.868 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-10-11T10:47:57.868 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ seq 0 3 2023-10-11T10:47:57.869 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-10-11T10:47:57.869 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-10-11T10:47:57.869 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1:0 2023-10-11T10:47:57.869 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T10:47:57.869 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance 2023-10-11T10:47:57.870 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-10-11T10:47:57.870 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local val=cluster1:0 2023-10-11T10:47:57.870 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_var_name=cluster 2023-10-11T10:47:57.870 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance_var_name=instance 2023-10-11T10:47:57.870 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ cluster=cluster1 2023-10-11T10:47:57.871 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ instance=0 2023-10-11T10:47:57.871 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' 0 = cluster1:0 ']' 2023-10-11T10:47:57.871 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval cluster=cluster1 2023-10-11T10:47:57.871 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T10:47:57.871 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval instance=0 2023-10-11T10:47:57.873 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=0 2023-10-11T10:47:57.873 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-10-11T10:47:57.873 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local local_cluster=cluster1:0 2023-10-11T10:47:57.874 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster=cluster1 2023-10-11T10:47:57.874 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance 2023-10-11T10:47:57.874 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-10-11T10:47:57.874 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local val=cluster1:0 2023-10-11T10:47:57.874 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster_var_name=local_cluster 2023-10-11T10:47:57.874 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance_var_name=instance 2023-10-11T10:47:57.875 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T10:47:57.875 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=0 2023-10-11T10:47:57.875 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ '[' 0 = cluster1:0 ']' 2023-10-11T10:47:57.875 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval local_cluster=cluster1 2023-10-11T10:47:57.875 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ local_cluster=cluster1 2023-10-11T10:47:57.876 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval instance=0 2023-10-11T10:47:57.876 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ instance=0 2023-10-11T10:47:57.876 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-10-11T10:47:57.889 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T10:47:57.890 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T10:47:57.890 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T10:47:57.890 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-10-11T10:47:57.990 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-10-11T10:47:57.990 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd mirror status : get status for rbd mirror 2023-10-11T10:47:57.990 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:admin_socket: invalid command 2023-10-11T10:47:57.993 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-10-11T10:47:57.993 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-10-11T10:47:57.993 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1:1 2023-10-11T10:47:57.994 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T10:47:57.994 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance 2023-10-11T10:47:57.994 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-10-11T10:47:57.994 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local val=cluster1:1 2023-10-11T10:47:57.994 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_var_name=cluster 2023-10-11T10:47:57.995 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance_var_name=instance 2023-10-11T10:47:57.995 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ cluster=cluster1 2023-10-11T10:47:57.995 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ instance=1 2023-10-11T10:47:57.995 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' 1 = cluster1:1 ']' 2023-10-11T10:47:57.995 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval cluster=cluster1 2023-10-11T10:47:57.996 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T10:47:57.996 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval instance=1 2023-10-11T10:47:57.996 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=1 2023-10-11T10:47:57.996 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-10-11T10:47:57.996 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local local_cluster=cluster1:1 2023-10-11T10:47:57.996 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster=cluster1 2023-10-11T10:47:57.997 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance 2023-10-11T10:47:57.997 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-10-11T10:47:57.997 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local val=cluster1:1 2023-10-11T10:47:57.997 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster_var_name=local_cluster 2023-10-11T10:47:57.997 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance_var_name=instance 2023-10-11T10:47:57.997 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T10:47:57.998 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=1 2023-10-11T10:47:57.998 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ '[' 1 = cluster1:1 ']' 2023-10-11T10:47:57.998 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval local_cluster=cluster1 2023-10-11T10:47:57.998 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ local_cluster=cluster1 2023-10-11T10:47:57.998 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval instance=1 2023-10-11T10:47:57.999 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ instance=1 2023-10-11T10:47:57.999 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-10-11T10:47:58.012 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T10:47:58.012 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T10:47:58.012 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T10:47:58.013 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-10-11T10:47:58.115 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ return 0 2023-10-11T10:47:58.116 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ status_result=' "state": "Replaying"' 2023-10-11T10:47:58.116 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ echo ' "state": "Replaying"' 2023-10-11T10:47:58.116 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ grep -i Replaying 2023-10-11T10:47:58.117 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: "state": "Replaying" 2023-10-11T10:47:58.118 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ current_state=started 2023-10-11T10:47:58.118 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test started = started 2023-10-11T10:47:58.119 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T10:47:58.119 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2023-10-11T10:47:58.119 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local local_cluster=cluster1 2023-10-11T10:47:58.119 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:47:58.119 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:47:58.120 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:47:58.120 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' snapshot = journal ']' 2023-10-11T10:47:58.120 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' snapshot = snapshot ']' 2023-10-11T10:47:58.120 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2023-10-11T10:47:58.120 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local local_cluster=cluster1 2023-10-11T10:47:58.120 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:47:58.121 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:47:58.121 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:47:58.121 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ mkfname cluster2-mirror-test.status 2023-10-11T10:47:58.121 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ echo cluster2-mirror-test.status 2023-10-11T10:47:58.121 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -e 's|[/ ]|_|g' 2023-10-11T10:47:58.121 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local status_log=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.status 2023-10-11T10:47:58.123 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ mkfname cluster1-mirror-test.status 2023-10-11T10:47:58.123 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ echo cluster1-mirror-test.status 2023-10-11T10:47:58.123 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -e 's|[/ ]|_|g' 2023-10-11T10:47:58.123 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local local_status_log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.status 2023-10-11T10:47:58.123 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ mirror_image_snapshot cluster2 mirror test 2023-10-11T10:47:58.123 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:47:58.124 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:47:58.124 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:47:58.124 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2023-10-11T10:47:58.497 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2023-10-11T10:47:58.498 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2023-10-11T10:47:58.498 INFO:tasks.rbd_mirror_thrash:waiting for 6 secs before reviving daemons 2023-10-11T10:47:58.498 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi111.stderr:2023-10-11T10:47:58.498+0000 7f8b4bd3e700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 106568) UID: 1000 2023-10-11T10:47:59.854 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:Snapshot ID: 17 2023-10-11T10:47:59.876 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.status 2023-10-11T10:47:59.877 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:47:59.877 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:47:59.877 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:47:59.877 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.status 2023-10-11T10:47:59.878 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T10:47:59.884 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2023-10-11T10:47:59.939 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-10-11T10:47:59.942 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snapshot_id=17 2023-10-11T10:47:59.942 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T10:47:59.942 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T10:47:59.942 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.2 2023-10-11T10:48:00.143 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.status 2023-10-11T10:48:00.143 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T10:48:00.143 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:48:00.144 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:48:00.144 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.status 2023-10-11T10:48:00.144 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-10-11T10:48:00.144 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T10:48:00.190 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T10:48:00.190 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T10:48:00.191 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:-:1.1: Document is empty 2023-10-11T10:48:00.191 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: 2023-10-11T10:48:00.191 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:^ 2023-10-11T10:48:00.192 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id= 2023-10-11T10:48:00.192 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 17 = '' 2023-10-11T10:48:00.192 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T10:48:00.193 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.4 2023-10-11T10:48:00.593 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.status 2023-10-11T10:48:00.593 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T10:48:00.594 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:48:00.594 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:48:00.594 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.status 2023-10-11T10:48:00.594 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-10-11T10:48:00.594 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T10:48:00.649 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T10:48:00.650 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T10:48:00.651 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:-:1.1: Document is empty 2023-10-11T10:48:00.651 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: 2023-10-11T10:48:00.651 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:^ 2023-10-11T10:48:00.652 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id= 2023-10-11T10:48:00.652 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 17 = '' 2023-10-11T10:48:00.652 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T10:48:00.653 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.8 2023-10-11T10:48:01.453 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.status 2023-10-11T10:48:01.453 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T10:48:01.454 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:48:01.454 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:48:01.454 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.status 2023-10-11T10:48:01.454 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-10-11T10:48:01.454 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T10:48:01.507 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T10:48:01.507 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T10:48:01.509 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:-:1.1: Document is empty 2023-10-11T10:48:01.509 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: 2023-10-11T10:48:01.509 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:^ 2023-10-11T10:48:01.510 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id= 2023-10-11T10:48:01.510 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 17 = '' 2023-10-11T10:48:01.510 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T10:48:01.510 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 1.6 2023-10-11T10:48:03.111 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.status 2023-10-11T10:48:03.111 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T10:48:03.111 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:48:03.111 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:48:03.112 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.status 2023-10-11T10:48:03.112 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-10-11T10:48:03.112 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T10:48:03.158 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T10:48:03.161 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id=17 2023-10-11T10:48:03.161 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 17 = 17 2023-10-11T10:48:03.161 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T10:48:03.161 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_snap_present cluster1 mirror test snap4 2023-10-11T10:48:03.161 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T10:48:03.161 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:48:03.162 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:48:03.162 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap4 2023-10-11T10:48:03.162 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T10:48:03.162 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-10-11T10:48:03.162 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 1 2023-10-11T10:48:04.161 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 info mirror/test@snap4 2023-10-11T10:48:04.209 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:rbd image 'test': 2023-10-11T10:48:04.209 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: size 512 MiB in 128 objects 2023-10-11T10:48:04.210 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: order 22 (4 MiB objects) 2023-10-11T10:48:04.210 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: snapshot_count: 8 2023-10-11T10:48:04.210 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: id: 10afefb6b972 2023-10-11T10:48:04.210 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: block_name_prefix: rbd_data.10afefb6b972 2023-10-11T10:48:04.210 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: format: 2 2023-10-11T10:48:04.211 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-10-11T10:48:04.211 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: op_features: 2023-10-11T10:48:04.211 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: flags: 2023-10-11T10:48:04.211 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: create_timestamp: Wed Oct 11 10:44:53 2023 2023-10-11T10:48:04.211 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: access_timestamp: Wed Oct 11 10:46:25 2023 2023-10-11T10:48:04.211 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: modify_timestamp: Wed Oct 11 10:44:53 2023 2023-10-11T10:48:04.212 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: protected: False 2023-10-11T10:48:04.212 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring state: enabled 2023-10-11T10:48:04.212 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring mode: snapshot 2023-10-11T10:48:04.212 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring global id: 3dd93db4-c9cc-462c-8b49-007eba9a90d2 2023-10-11T10:48:04.212 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring primary: false 2023-10-11T10:48:04.213 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T10:48:04.214 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n snap3-clean ']' 2023-10-11T10:48:04.214 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ compare_image_snaps mirror test snap3-clean 2023-10-11T10:48:04.214 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:48:04.214 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:48:04.214 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap3-clean 2023-10-11T10:48:04.215 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local ret=0 2023-10-11T10:48:04.215 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local rmt_export=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export 2023-10-11T10:48:04.215 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local loc_export=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:48:04.215 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rm -f /tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:48:04.215 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 -p mirror export test@snap3-clean /tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export 2023-10-11T10:48:04.499 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2023-10-11T10:48:04.499 INFO:teuthology.orchestra.run:waiting for 600 2023-10-11T10:48:04.499 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2023-10-11T10:48:04.500 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2023-10-11T10:48:04.500 DEBUG:teuthology.orchestra.run.smithi111:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2023-10-11T10:48:04.503 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2023-10-11T10:48:04.503 INFO:tasks.rbd_mirror_thrash:waiting for 104 secs before thrashing 2023-10-11T10:48:04.806 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-10-11T10:48:04.816 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 -p mirror export test@snap3-clean /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:48:05.537 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-10-11T10:48:05.546 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ cmp /tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:48:05.761 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rm -f /tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:48:05.902 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T10:48:05.902 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ compare_image_snaps mirror test snap4 2023-10-11T10:48:05.902 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:48:05.903 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:48:05.903 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap4 2023-10-11T10:48:05.903 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local ret=0 2023-10-11T10:48:05.903 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local rmt_export=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export 2023-10-11T10:48:05.903 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local loc_export=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:48:05.904 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rm -f /tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:48:05.904 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 -p mirror export test@snap4 /tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export 2023-10-11T10:48:06.493 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-10-11T10:48:06.499 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 -p mirror export test@snap4 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:48:07.153 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-10-11T10:48:07.160 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ cmp /tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:48:07.376 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rm -f /tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:48:07.517 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T10:48:07.517 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ clean_snap_name=snap4-clean 2023-10-11T10:48:07.518 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ create_snap cluster2 mirror test snap4-clean 2023-10-11T10:48:07.518 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:48:07.518 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:48:07.518 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:48:07.519 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap4-clean 2023-10-11T10:48:07.519 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap4-clean --debug-rbd=20 --debug-journaler=20 2023-10-11T10:48:07.872 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 10` 2023-10-11T10:48:07.873 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ stress_write_image cluster2 mirror test 2023-10-11T10:48:07.873 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:48:07.873 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:48:07.873 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:48:07.873 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-10-11T10:48:07.874 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local duration=7 2023-10-11T10:48:07.874 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ set +e 2023-10-11T10:48:07.874 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2023-10-11T10:48:08.095 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:100 IOs, 20508672 bytes 2023-10-11T10:48:08.260 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:200 IOs, 78533632 bytes 2023-10-11T10:48:08.400 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:300 IOs, 87786496 bytes 2023-10-11T10:48:08.577 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:400 IOs, 141883392 bytes 2023-10-11T10:48:08.801 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:500 IOs, 229999616 bytes 2023-10-11T10:48:08.877 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:600 IOs, 230610944 bytes 2023-10-11T10:48:08.995 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:700 IOs, 279881728 bytes 2023-10-11T10:48:09.100 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:800 IOs, 294170624 bytes 2023-10-11T10:48:09.350 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:900 IOs, 388139008 bytes 2023-10-11T10:48:09.496 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1000 IOs, 404142080 bytes 2023-10-11T10:48:09.798 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1100 IOs, 518913024 bytes 2023-10-11T10:48:09.896 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1200 IOs, 526423040 bytes 2023-10-11T10:48:10.090 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1300 IOs, 586691584 bytes 2023-10-11T10:48:10.273 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1400 IOs, 641396736 bytes 2023-10-11T10:48:10.355 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1500 IOs, 664634368 bytes 2023-10-11T10:48:10.498 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1600 IOs, 694647808 bytes 2023-10-11T10:48:10.588 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1700 IOs, 708868096 bytes 2023-10-11T10:48:10.726 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1800 IOs, 741549056 bytes 2023-10-11T10:48:10.820 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1900 IOs, 778336256 bytes 2023-10-11T10:48:10.999 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2000 IOs, 841730048 bytes 2023-10-11T10:48:11.154 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2100 IOs, 880962560 bytes 2023-10-11T10:48:11.401 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2200 IOs, 957181952 bytes 2023-10-11T10:48:11.510 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2300 IOs, 975778816 bytes 2023-10-11T10:48:11.605 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2400 IOs, 989733888 bytes 2023-10-11T10:48:11.700 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2500 IOs, 1001226240 bytes 2023-10-11T10:48:11.859 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2600 IOs, 1047839744 bytes 2023-10-11T10:48:11.964 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2700 IOs, 1065940992 bytes 2023-10-11T10:48:12.093 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2800 IOs, 1093665792 bytes 2023-10-11T10:48:12.229 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2900 IOs, 1124334592 bytes 2023-10-11T10:48:12.429 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3000 IOs, 1193244672 bytes 2023-10-11T10:48:12.566 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3100 IOs, 1246531584 bytes 2023-10-11T10:48:12.751 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3200 IOs, 1284747264 bytes 2023-10-11T10:48:12.886 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3300 IOs, 1311093760 bytes 2023-10-11T10:48:13.163 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3400 IOs, 1425718272 bytes 2023-10-11T10:48:13.428 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3500 IOs, 1508709376 bytes 2023-10-11T10:48:13.646 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3600 IOs, 1587566592 bytes 2023-10-11T10:48:13.787 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3700 IOs, 1643566080 bytes 2023-10-11T10:48:14.030 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3800 IOs, 1710431232 bytes 2023-10-11T10:48:14.145 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3900 IOs, 1735792640 bytes 2023-10-11T10:48:14.300 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4000 IOs, 1776543744 bytes 2023-10-11T10:48:14.415 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4100 IOs, 1798130688 bytes 2023-10-11T10:48:14.538 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4200 IOs, 1823059968 bytes 2023-10-11T10:48:14.642 INFO:tasks.ceph.cluster2.osd.2.smithi111.stderr:2023-10-11T10:48:14.638+0000 7fa35bd9b700 -1 reset not still connected to 0x5563f778eb60 2023-10-11T10:48:14.642 INFO:tasks.ceph.cluster2.osd.2.smithi111.stderr:2023-10-11T10:48:14.638+0000 7fa35bd9b700 -1 reset not still connected to 0x5563fabe0820 2023-10-11T10:48:14.642 INFO:tasks.ceph.cluster2.osd.2.smithi111.stderr:2023-10-11T10:48:14.638+0000 7fa35bd9b700 -1 reset not still connected to 0x5563fabe0a90 2023-10-11T10:48:14.642 INFO:tasks.ceph.cluster2.osd.2.smithi111.stderr:2023-10-11T10:48:14.638+0000 7fa35bd9b700 -1 reset not still connected to 0x5563fabe0d00 2023-10-11T10:48:14.701 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4300 IOs, 1867897856 bytes 2023-10-11T10:48:14.765 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4400 IOs, 1868507136 bytes 2023-10-11T10:48:14.879 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ error_code=124 2023-10-11T10:48:14.879 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ set -e 2023-10-11T10:48:14.879 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' 124 -eq 124 ']' 2023-10-11T10:48:14.879 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T10:48:14.880 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2023-10-11T10:48:14.880 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T10:48:14.880 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:48:14.880 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:48:14.880 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local state_pattern=up+replaying 2023-10-11T10:48:14.881 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local description_pattern= 2023-10-11T10:48:14.881 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local service_pattern= 2023-10-11T10:48:14.881 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-10-11T10:48:14.881 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 1 2023-10-11T10:48:15.880 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2023-10-11T10:48:15.880 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T10:48:15.881 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:48:15.881 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:48:15.881 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local state_pattern=up+replaying 2023-10-11T10:48:15.881 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local description_pattern= 2023-10-11T10:48:15.881 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local service_pattern= 2023-10-11T10:48:15.882 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ mkfname cluster1-mirror-test.mirror_status 2023-10-11T10:48:15.882 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ echo cluster1-mirror-test.mirror_status 2023-10-11T10:48:15.882 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -e 's|[/ ]|_|g' 2023-10-11T10:48:15.883 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local status_log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.mirror_status 2023-10-11T10:48:15.883 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ CEPH_ARGS= 2023-10-11T10:48:15.883 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2023-10-11T10:48:15.883 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ tee /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.mirror_status 2023-10-11T10:48:15.933 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:test: 2023-10-11T10:48:15.933 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: global_id: 3dd93db4-c9cc-462c-8b49-007eba9a90d2 2023-10-11T10:48:15.934 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: state: up+replaying 2023-10-11T10:48:15.934 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: description: replaying, {"bytes_per_second":17687040.0,"bytes_per_snapshot":513039872.0,"last_snapshot_bytes":530611200,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1697021235,"remote_snapshot_timestamp":1697021235,"replay_state":"idle"} 2023-10-11T10:48:15.934 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: service: mirror.1 on smithi111 2023-10-11T10:48:15.934 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: last_update: 2023-10-11 10:47:51 2023-10-11T10:48:15.934 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: peer_sites: 2023-10-11T10:48:15.934 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: name: cluster2 2023-10-11T10:48:15.935 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: state: up+stopped 2023-10-11T10:48:15.935 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: description: local image is primary 2023-10-11T10:48:15.935 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: last_update: 2023-10-11 10:47:51 2023-10-11T10:48:15.938 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.mirror_status 2023-10-11T10:48:15.939 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: state: up+replaying 2023-10-11T10:48:15.939 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ grep '^ description: .*' /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.mirror_status 2023-10-11T10:48:15.940 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: description: replaying, {"bytes_per_second":17687040.0,"bytes_per_snapshot":513039872.0,"last_snapshot_bytes":530611200,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1697021235,"remote_snapshot_timestamp":1697021235,"replay_state":"idle"} 2023-10-11T10:48:15.941 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n '' ']' 2023-10-11T10:48:15.941 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ echo up+replaying 2023-10-11T10:48:15.941 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ grep '^up+' 2023-10-11T10:48:15.941 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:up+replaying 2023-10-11T10:48:15.942 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ grep 'service: *mirror..* on ' /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.mirror_status 2023-10-11T10:48:15.942 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: service: mirror.1 on smithi111 2023-10-11T10:48:15.943 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.mirror_status 2023-10-11T10:48:15.944 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local 'last_update=2023-10-11 10:47:51' 2023-10-11T10:48:15.944 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2023-10-11 10:47:51' 2023-10-11T10:48:15.945 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T10:48:15.945 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:48:15.945 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:48:15.945 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local state_pattern=up+replaying 2023-10-11T10:48:15.945 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local 'prev_last_update=2023-10-11 10:47:51' 2023-10-11T10:48:15.945 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ mkfname cluster1-mirror.mirror_status 2023-10-11T10:48:15.946 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ echo cluster1-mirror.mirror_status 2023-10-11T10:48:15.946 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -e 's|[/ ]|_|g' 2023-10-11T10:48:15.946 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local status_log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror.mirror_status 2023-10-11T10:48:15.947 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2023-10-11T10:48:15.982 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd: failed to query services: (13) Permission denied 2023-10-11T10:48:16.004 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local last_update state 2023-10-11T10:48:16.005 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2023-10-11T10:48:16.007 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ last_update='2023-10-11 10:47:51' 2023-10-11T10:48:16.007 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2023-10-11T10:48:16.009 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ state=up+replaying 2023-10-11T10:48:16.009 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ echo up+replaying 2023-10-11T10:48:16.009 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ grep up+replaying 2023-10-11T10:48:16.010 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:up+replaying 2023-10-11T10:48:16.011 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T10:48:16.011 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T10:48:16.011 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ snap_name=snap5 2023-10-11T10:48:16.011 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ create_snap cluster2 mirror test snap5 2023-10-11T10:48:16.012 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:48:16.012 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:48:16.012 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:48:16.012 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap5 2023-10-11T10:48:16.012 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap5 --debug-rbd=20 --debug-journaler=20 2023-10-11T10:48:43.506 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.6286768504849554) > weight (0.25) 2023-10-11T10:48:43.507 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.1 2023-10-11T10:48:43.508 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Sent signal 15 2023-10-11T10:48:43.509 INFO:tasks.rbd_mirror_thrash:waiting for 8 secs before reviving daemons 2023-10-11T10:48:43.510 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi111.stderr:2023-10-11T10:48:43.508+0000 7f37dd4d6700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 106662) UID: 1000 2023-10-11T10:48:45.886 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_replay_started cluster1 mirror test 2023-10-11T10:48:45.886 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T10:48:45.886 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:48:45.886 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:48:45.887 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2023-10-11T10:48:45.887 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T10:48:45.887 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:48:45.887 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:48:45.887 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local state=started 2023-10-11T10:48:45.887 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T10:48:45.888 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-10-11T10:48:45.888 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 1 2023-10-11T10:48:46.888 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test_image_replay_state cluster1 mirror test started 2023-10-11T10:48:46.888 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T10:48:46.888 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:48:46.888 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:48:46.889 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local test_state=started 2023-10-11T10:48:46.889 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local status_result 2023-10-11T10:48:46.889 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_state=stopped 2023-10-11T10:48:46.889 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2023-10-11T10:48:46.889 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_instance=cluster1 2023-10-11T10:48:46.890 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ grep -i state 2023-10-11T10:48:46.890 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T10:48:46.890 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T10:48:46.890 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance=cluster1 2023-10-11T10:48:46.890 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local loop_instance 2023-10-11T10:48:46.891 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-10-11T10:48:46.891 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sleep 0 2023-10-11T10:48:46.893 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-10-11T10:48:46.893 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ seq 0 3 2023-10-11T10:48:46.893 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-10-11T10:48:46.894 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-10-11T10:48:46.894 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1:0 2023-10-11T10:48:46.894 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T10:48:46.894 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance 2023-10-11T10:48:46.894 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-10-11T10:48:46.895 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local val=cluster1:0 2023-10-11T10:48:46.895 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_var_name=cluster 2023-10-11T10:48:46.895 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance_var_name=instance 2023-10-11T10:48:46.895 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ cluster=cluster1 2023-10-11T10:48:46.896 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ instance=0 2023-10-11T10:48:46.896 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' 0 = cluster1:0 ']' 2023-10-11T10:48:46.896 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval cluster=cluster1 2023-10-11T10:48:46.896 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T10:48:46.897 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval instance=0 2023-10-11T10:48:46.897 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=0 2023-10-11T10:48:46.897 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-10-11T10:48:46.899 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local local_cluster=cluster1:0 2023-10-11T10:48:46.899 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster=cluster1 2023-10-11T10:48:46.899 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance 2023-10-11T10:48:46.899 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-10-11T10:48:46.900 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local val=cluster1:0 2023-10-11T10:48:46.900 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster_var_name=local_cluster 2023-10-11T10:48:46.900 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance_var_name=instance 2023-10-11T10:48:46.900 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T10:48:46.900 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=0 2023-10-11T10:48:46.900 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ '[' 0 = cluster1:0 ']' 2023-10-11T10:48:46.901 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval local_cluster=cluster1 2023-10-11T10:48:46.901 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ local_cluster=cluster1 2023-10-11T10:48:46.901 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval instance=0 2023-10-11T10:48:46.901 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ instance=0 2023-10-11T10:48:46.901 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-10-11T10:48:46.913 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T10:48:46.913 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T10:48:46.913 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T10:48:46.913 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-10-11T10:48:47.013 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-10-11T10:48:47.013 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd mirror status : get status for rbd mirror 2023-10-11T10:48:47.013 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:admin_socket: invalid command 2023-10-11T10:48:47.016 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-10-11T10:48:47.016 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-10-11T10:48:47.016 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1:1 2023-10-11T10:48:47.016 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T10:48:47.017 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance 2023-10-11T10:48:47.017 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-10-11T10:48:47.017 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local val=cluster1:1 2023-10-11T10:48:47.017 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_var_name=cluster 2023-10-11T10:48:47.017 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance_var_name=instance 2023-10-11T10:48:47.018 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ cluster=cluster1 2023-10-11T10:48:47.018 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ instance=1 2023-10-11T10:48:47.018 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' 1 = cluster1:1 ']' 2023-10-11T10:48:47.018 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval cluster=cluster1 2023-10-11T10:48:47.018 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T10:48:47.018 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval instance=1 2023-10-11T10:48:47.019 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=1 2023-10-11T10:48:47.019 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-10-11T10:48:47.019 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local local_cluster=cluster1:1 2023-10-11T10:48:47.019 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster=cluster1 2023-10-11T10:48:47.019 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance 2023-10-11T10:48:47.019 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-10-11T10:48:47.020 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local val=cluster1:1 2023-10-11T10:48:47.020 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster_var_name=local_cluster 2023-10-11T10:48:47.020 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance_var_name=instance 2023-10-11T10:48:47.020 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T10:48:47.020 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=1 2023-10-11T10:48:47.021 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ '[' 1 = cluster1:1 ']' 2023-10-11T10:48:47.021 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval local_cluster=cluster1 2023-10-11T10:48:47.021 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ local_cluster=cluster1 2023-10-11T10:48:47.021 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval instance=1 2023-10-11T10:48:47.021 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ instance=1 2023-10-11T10:48:47.021 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-10-11T10:48:47.035 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T10:48:47.035 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T10:48:47.035 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T10:48:47.035 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-10-11T10:48:47.139 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ return 0 2023-10-11T10:48:47.139 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ status_result=' "state": "Replaying"' 2023-10-11T10:48:47.140 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ echo ' "state": "Replaying"' 2023-10-11T10:48:47.140 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ grep -i Replaying 2023-10-11T10:48:47.140 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: "state": "Replaying" 2023-10-11T10:48:47.142 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ current_state=started 2023-10-11T10:48:47.142 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test started = started 2023-10-11T10:48:47.142 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T10:48:47.143 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2023-10-11T10:48:47.143 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local local_cluster=cluster1 2023-10-11T10:48:47.143 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:48:47.143 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:48:47.143 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:48:47.144 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' snapshot = journal ']' 2023-10-11T10:48:47.144 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' snapshot = snapshot ']' 2023-10-11T10:48:47.144 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2023-10-11T10:48:47.144 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local local_cluster=cluster1 2023-10-11T10:48:47.144 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:48:47.144 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:48:47.145 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:48:47.145 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ mkfname cluster2-mirror-test.status 2023-10-11T10:48:47.145 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ echo cluster2-mirror-test.status 2023-10-11T10:48:47.145 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -e 's|[/ ]|_|g' 2023-10-11T10:48:47.145 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local status_log=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.status 2023-10-11T10:48:47.147 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ mkfname cluster1-mirror-test.status 2023-10-11T10:48:47.147 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ echo cluster1-mirror-test.status 2023-10-11T10:48:47.147 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -e 's|[/ ]|_|g' 2023-10-11T10:48:47.147 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local local_status_log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.status 2023-10-11T10:48:47.147 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ mirror_image_snapshot cluster2 mirror test 2023-10-11T10:48:47.148 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:48:47.148 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:48:47.148 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:48:47.148 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2023-10-11T10:48:48.897 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:Snapshot ID: 21 2023-10-11T10:48:48.923 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.status 2023-10-11T10:48:48.923 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:48:48.923 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:48:48.923 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:48:48.923 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.status 2023-10-11T10:48:48.925 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2023-10-11T10:48:48.925 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T10:48:48.983 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-10-11T10:48:48.986 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snapshot_id=21 2023-10-11T10:48:48.986 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T10:48:48.987 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T10:48:48.987 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.2 2023-10-11T10:48:49.188 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.status 2023-10-11T10:48:49.188 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T10:48:49.188 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:48:49.188 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:48:49.188 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.status 2023-10-11T10:48:49.189 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-10-11T10:48:49.189 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T10:48:49.234 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T10:48:49.235 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T10:48:49.236 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:-:1.1: Document is empty 2023-10-11T10:48:49.236 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: 2023-10-11T10:48:49.236 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:^ 2023-10-11T10:48:49.237 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id= 2023-10-11T10:48:49.237 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 21 = '' 2023-10-11T10:48:49.237 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T10:48:49.237 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.4 2023-10-11T10:48:49.638 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.status 2023-10-11T10:48:49.638 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T10:48:49.638 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:48:49.639 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:48:49.639 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.status 2023-10-11T10:48:49.639 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-10-11T10:48:49.639 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T10:48:49.684 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T10:48:49.685 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T10:48:49.686 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:-:1.1: Document is empty 2023-10-11T10:48:49.687 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: 2023-10-11T10:48:49.687 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:^ 2023-10-11T10:48:49.687 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id= 2023-10-11T10:48:49.687 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 21 = '' 2023-10-11T10:48:49.687 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T10:48:49.687 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.8 2023-10-11T10:48:50.488 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.status 2023-10-11T10:48:50.488 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T10:48:50.489 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:48:50.489 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:48:50.489 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.status 2023-10-11T10:48:50.489 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-10-11T10:48:50.489 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T10:48:50.574 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T10:48:50.575 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T10:48:50.576 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:-:1.1: Document is empty 2023-10-11T10:48:50.576 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: 2023-10-11T10:48:50.577 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:^ 2023-10-11T10:48:50.577 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id= 2023-10-11T10:48:50.577 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 21 = '' 2023-10-11T10:48:50.578 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T10:48:50.578 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 1.6 2023-10-11T10:48:51.510 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.1 2023-10-11T10:48:51.510 INFO:teuthology.orchestra.run:waiting for 600 2023-10-11T10:48:51.511 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.1 2023-10-11T10:48:51.511 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2023-10-11T10:48:51.511 DEBUG:teuthology.orchestra.run.smithi111:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2023-10-11T10:48:51.514 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2023-10-11T10:48:51.514 INFO:tasks.rbd_mirror_thrash:waiting for 113 secs before thrashing 2023-10-11T10:48:52.178 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.status 2023-10-11T10:48:52.178 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T10:48:52.179 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:48:52.179 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:48:52.179 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.status 2023-10-11T10:48:52.179 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-10-11T10:48:52.179 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T10:48:52.225 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T10:48:52.227 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id=21 2023-10-11T10:48:52.227 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 21 = 21 2023-10-11T10:48:52.228 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T10:48:52.228 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_snap_present cluster1 mirror test snap5 2023-10-11T10:48:52.228 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T10:48:52.228 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:48:52.228 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:48:52.229 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap5 2023-10-11T10:48:52.229 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T10:48:52.229 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-10-11T10:48:52.229 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 1 2023-10-11T10:48:53.228 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 info mirror/test@snap5 2023-10-11T10:48:53.479 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:rbd image 'test': 2023-10-11T10:48:53.479 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: size 512 MiB in 128 objects 2023-10-11T10:48:53.479 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: order 22 (4 MiB objects) 2023-10-11T10:48:53.480 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: snapshot_count: 10 2023-10-11T10:48:53.480 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: id: 10afefb6b972 2023-10-11T10:48:53.480 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: block_name_prefix: rbd_data.10afefb6b972 2023-10-11T10:48:53.480 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: format: 2 2023-10-11T10:48:53.480 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-10-11T10:48:53.480 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: op_features: 2023-10-11T10:48:53.481 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: flags: 2023-10-11T10:48:53.481 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: create_timestamp: Wed Oct 11 10:44:53 2023 2023-10-11T10:48:53.481 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: access_timestamp: Wed Oct 11 10:48:04 2023 2023-10-11T10:48:53.481 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: modify_timestamp: Wed Oct 11 10:44:53 2023 2023-10-11T10:48:53.481 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: protected: False 2023-10-11T10:48:53.482 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring state: enabled 2023-10-11T10:48:53.482 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring mode: snapshot 2023-10-11T10:48:53.482 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring global id: 3dd93db4-c9cc-462c-8b49-007eba9a90d2 2023-10-11T10:48:53.482 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring primary: false 2023-10-11T10:48:53.483 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T10:48:53.483 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n snap4-clean ']' 2023-10-11T10:48:53.484 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ compare_image_snaps mirror test snap4-clean 2023-10-11T10:48:53.484 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:48:53.484 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:48:53.484 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap4-clean 2023-10-11T10:48:53.484 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local ret=0 2023-10-11T10:48:53.484 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local rmt_export=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export 2023-10-11T10:48:53.485 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local loc_export=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:48:53.485 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rm -f /tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:48:53.485 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 -p mirror export test@snap4-clean /tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export 2023-10-11T10:48:54.077 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-10-11T10:48:54.083 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 -p mirror export test@snap4-clean /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:48:54.808 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-10-11T10:48:54.818 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ cmp /tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:48:55.031 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rm -f /tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:48:55.172 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T10:48:55.172 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ compare_image_snaps mirror test snap5 2023-10-11T10:48:55.173 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:48:55.173 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:48:55.173 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap5 2023-10-11T10:48:55.173 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local ret=0 2023-10-11T10:48:55.173 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local rmt_export=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export 2023-10-11T10:48:55.173 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local loc_export=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:48:55.174 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rm -f /tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:48:55.174 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 -p mirror export test@snap5 /tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export 2023-10-11T10:48:55.739 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-10-11T10:48:55.749 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 -p mirror export test@snap5 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:48:56.520 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-10-11T10:48:56.527 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ cmp /tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:48:56.746 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rm -f /tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:48:56.888 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T10:48:56.888 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ clean_snap_name=snap5-clean 2023-10-11T10:48:56.888 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ create_snap cluster2 mirror test snap5-clean 2023-10-11T10:48:56.888 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:48:56.889 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:48:56.889 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:48:56.889 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap5-clean 2023-10-11T10:48:56.889 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap5-clean --debug-rbd=20 --debug-journaler=20 2023-10-11T10:48:57.219 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 10` 2023-10-11T10:48:57.219 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ stress_write_image cluster2 mirror test 2023-10-11T10:48:57.219 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:48:57.219 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:48:57.220 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:48:57.220 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-10-11T10:48:57.220 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local duration=12 2023-10-11T10:48:57.221 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ set +e 2023-10-11T10:48:57.221 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2023-10-11T10:48:57.519 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:100 IOs, 54235136 bytes 2023-10-11T10:48:57.774 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:200 IOs, 126788608 bytes 2023-10-11T10:48:57.961 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:300 IOs, 186394624 bytes 2023-10-11T10:48:58.109 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:400 IOs, 230611968 bytes 2023-10-11T10:48:58.203 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:500 IOs, 243235840 bytes 2023-10-11T10:48:58.342 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:600 IOs, 281601024 bytes 2023-10-11T10:48:58.496 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:700 IOs, 325562368 bytes 2023-10-11T10:48:58.609 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:800 IOs, 344306688 bytes 2023-10-11T10:48:58.868 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:900 IOs, 440428544 bytes 2023-10-11T10:48:58.944 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1000 IOs, 456064000 bytes 2023-10-11T10:48:59.168 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1100 IOs, 526100480 bytes 2023-10-11T10:48:59.307 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1200 IOs, 559010816 bytes 2023-10-11T10:48:59.442 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1300 IOs, 593082368 bytes 2023-10-11T10:48:59.649 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1400 IOs, 661210112 bytes 2023-10-11T10:48:59.811 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1500 IOs, 710063104 bytes 2023-10-11T10:48:59.977 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1600 IOs, 756319232 bytes 2023-10-11T10:49:00.105 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1700 IOs, 784350208 bytes 2023-10-11T10:49:00.243 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1800 IOs, 819335168 bytes 2023-10-11T10:49:00.329 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1900 IOs, 824487936 bytes 2023-10-11T10:49:00.408 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2000 IOs, 832058368 bytes 2023-10-11T10:49:00.531 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2100 IOs, 856505344 bytes 2023-10-11T10:49:00.691 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2200 IOs, 904857600 bytes 2023-10-11T10:49:00.863 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2300 IOs, 954916864 bytes 2023-10-11T10:49:01.019 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2400 IOs, 1001929728 bytes 2023-10-11T10:49:01.271 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2500 IOs, 1099233280 bytes 2023-10-11T10:49:01.396 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2600 IOs, 1120277504 bytes 2023-10-11T10:49:01.557 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2700 IOs, 1159351296 bytes 2023-10-11T10:49:01.818 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2800 IOs, 1253184512 bytes 2023-10-11T10:49:02.096 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2900 IOs, 1352170496 bytes 2023-10-11T10:49:02.293 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3000 IOs, 1415017472 bytes 2023-10-11T10:49:02.403 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3100 IOs, 1454388224 bytes 2023-10-11T10:49:02.724 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3200 IOs, 1505821696 bytes 2023-10-11T10:49:02.868 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3300 IOs, 1540793344 bytes 2023-10-11T10:49:03.071 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3400 IOs, 1607317504 bytes 2023-10-11T10:49:03.252 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3500 IOs, 1664479232 bytes 2023-10-11T10:49:03.405 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3600 IOs, 1709459456 bytes 2023-10-11T10:49:03.565 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3700 IOs, 1755963392 bytes 2023-10-11T10:49:03.724 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3800 IOs, 1803164672 bytes 2023-10-11T10:49:03.846 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3900 IOs, 1830629376 bytes 2023-10-11T10:49:03.926 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4000 IOs, 1832867840 bytes 2023-10-11T10:49:04.088 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4100 IOs, 1876666368 bytes 2023-10-11T10:49:04.342 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4200 IOs, 1966803968 bytes 2023-10-11T10:49:04.434 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4300 IOs, 1976497152 bytes 2023-10-11T10:49:04.584 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4400 IOs, 2008367104 bytes 2023-10-11T10:49:04.704 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4500 IOs, 2030481408 bytes 2023-10-11T10:49:04.836 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4600 IOs, 2061306880 bytes 2023-10-11T10:49:05.005 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4700 IOs, 2112945152 bytes 2023-10-11T10:49:05.194 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4800 IOs, 2169425920 bytes 2023-10-11T10:49:05.417 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4900 IOs, 2244258816 bytes 2023-10-11T10:49:05.600 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5000 IOs, 2305081344 bytes 2023-10-11T10:49:05.781 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5100 IOs, 2362299392 bytes 2023-10-11T10:49:05.922 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5200 IOs, 2411086848 bytes 2023-10-11T10:49:06.093 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5300 IOs, 2453982208 bytes 2023-10-11T10:49:06.202 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5400 IOs, 2472262656 bytes 2023-10-11T10:49:06.445 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5500 IOs, 2551808000 bytes 2023-10-11T10:49:06.676 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5600 IOs, 2635124736 bytes 2023-10-11T10:49:06.814 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5700 IOs, 2666253312 bytes 2023-10-11T10:49:06.964 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5800 IOs, 2729857024 bytes 2023-10-11T10:49:07.115 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5900 IOs, 2753574912 bytes 2023-10-11T10:49:07.252 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6000 IOs, 2782219264 bytes 2023-10-11T10:49:07.393 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6100 IOs, 2819713024 bytes 2023-10-11T10:49:07.550 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6200 IOs, 2867701760 bytes 2023-10-11T10:49:07.730 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6300 IOs, 2925082624 bytes 2023-10-11T10:49:07.978 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6400 IOs, 3019071488 bytes 2023-10-11T10:49:08.336 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6500 IOs, 3173238784 bytes 2023-10-11T10:49:08.527 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6600 IOs, 3229068288 bytes 2023-10-11T10:49:08.638 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6700 IOs, 3253740544 bytes 2023-10-11T10:49:08.759 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6800 IOs, 3281258496 bytes 2023-10-11T10:49:08.878 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6900 IOs, 3304200192 bytes 2023-10-11T10:49:08.963 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:7000 IOs, 3313714176 bytes 2023-10-11T10:49:09.230 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ error_code=124 2023-10-11T10:49:09.230 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ set -e 2023-10-11T10:49:09.230 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' 124 -eq 124 ']' 2023-10-11T10:49:09.231 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T10:49:09.231 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2023-10-11T10:49:09.231 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T10:49:09.231 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:49:09.231 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:49:09.232 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local state_pattern=up+replaying 2023-10-11T10:49:09.232 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local description_pattern= 2023-10-11T10:49:09.232 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local service_pattern= 2023-10-11T10:49:09.232 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-10-11T10:49:09.232 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 1 2023-10-11T10:49:10.232 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2023-10-11T10:49:10.232 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T10:49:10.232 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:49:10.232 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:49:10.233 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local state_pattern=up+replaying 2023-10-11T10:49:10.233 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local description_pattern= 2023-10-11T10:49:10.233 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local service_pattern= 2023-10-11T10:49:10.233 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ mkfname cluster1-mirror-test.mirror_status 2023-10-11T10:49:10.233 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ echo cluster1-mirror-test.mirror_status 2023-10-11T10:49:10.234 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -e 's|[/ ]|_|g' 2023-10-11T10:49:10.234 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local status_log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.mirror_status 2023-10-11T10:49:10.234 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ CEPH_ARGS= 2023-10-11T10:49:10.235 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2023-10-11T10:49:10.235 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ tee /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.mirror_status 2023-10-11T10:49:10.286 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:test: 2023-10-11T10:49:10.287 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: global_id: 3dd93db4-c9cc-462c-8b49-007eba9a90d2 2023-10-11T10:49:10.287 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: state: up+replaying 2023-10-11T10:49:10.287 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: description: replaying, {"bytes_per_second":9361783.470000001,"bytes_per_snapshot":502672384.0,"last_snapshot_bytes":522615808,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1697021278,"remote_snapshot_timestamp":1697021327,"replay_state":"syncing","seconds_until_synced":53,"syncing_percent":100,"syncing_snapshot_timestamp":1697021327} 2023-10-11T10:49:10.287 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: service: mirror.1 on smithi111 2023-10-11T10:49:10.287 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: last_update: 2023-10-11 10:48:51 2023-10-11T10:49:10.288 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: peer_sites: 2023-10-11T10:49:10.288 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: name: cluster2 2023-10-11T10:49:10.288 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: state: up+stopped 2023-10-11T10:49:10.288 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: description: local image is primary 2023-10-11T10:49:10.288 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: last_update: 2023-10-11 10:48:51 2023-10-11T10:49:10.291 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.mirror_status 2023-10-11T10:49:10.293 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: state: up+replaying 2023-10-11T10:49:10.293 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ grep '^ description: .*' /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.mirror_status 2023-10-11T10:49:10.294 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: description: replaying, {"bytes_per_second":9361783.470000001,"bytes_per_snapshot":502672384.0,"last_snapshot_bytes":522615808,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1697021278,"remote_snapshot_timestamp":1697021327,"replay_state":"syncing","seconds_until_synced":53,"syncing_percent":100,"syncing_snapshot_timestamp":1697021327} 2023-10-11T10:49:10.294 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n '' ']' 2023-10-11T10:49:10.294 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ echo up+replaying 2023-10-11T10:49:10.295 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ grep '^up+' 2023-10-11T10:49:10.295 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:up+replaying 2023-10-11T10:49:10.295 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ grep 'service: *mirror..* on ' /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.mirror_status 2023-10-11T10:49:10.296 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: service: mirror.1 on smithi111 2023-10-11T10:49:10.296 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.mirror_status 2023-10-11T10:49:10.298 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local 'last_update=2023-10-11 10:48:51' 2023-10-11T10:49:10.298 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2023-10-11 10:48:51' 2023-10-11T10:49:10.298 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T10:49:10.299 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:49:10.299 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:49:10.299 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local state_pattern=up+replaying 2023-10-11T10:49:10.299 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local 'prev_last_update=2023-10-11 10:48:51' 2023-10-11T10:49:10.299 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ mkfname cluster1-mirror.mirror_status 2023-10-11T10:49:10.300 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ echo cluster1-mirror.mirror_status 2023-10-11T10:49:10.300 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -e 's|[/ ]|_|g' 2023-10-11T10:49:10.300 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local status_log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror.mirror_status 2023-10-11T10:49:10.301 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2023-10-11T10:49:10.336 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd: failed to query services: (13) Permission denied 2023-10-11T10:49:10.359 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local last_update state 2023-10-11T10:49:10.359 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2023-10-11T10:49:10.361 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ last_update='2023-10-11 10:48:51' 2023-10-11T10:49:10.361 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2023-10-11T10:49:10.363 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ state=up+replaying 2023-10-11T10:49:10.363 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ echo up+replaying 2023-10-11T10:49:10.364 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ grep up+replaying 2023-10-11T10:49:10.364 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:up+replaying 2023-10-11T10:49:10.365 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T10:49:10.365 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T10:49:10.365 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ snap_name=snap6 2023-10-11T10:49:10.366 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ create_snap cluster2 mirror test snap6 2023-10-11T10:49:10.366 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:49:10.366 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:49:10.366 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:49:10.366 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap6 2023-10-11T10:49:10.367 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap6 --debug-rbd=20 --debug-journaler=20 2023-10-11T10:49:38.883 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_replay_started cluster1 mirror test 2023-10-11T10:49:38.884 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T10:49:38.884 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:49:38.884 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:49:38.884 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2023-10-11T10:49:38.885 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T10:49:38.885 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:49:38.885 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:49:38.885 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local state=started 2023-10-11T10:49:38.885 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T10:49:38.886 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-10-11T10:49:38.886 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 1 2023-10-11T10:49:39.885 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test_image_replay_state cluster1 mirror test started 2023-10-11T10:49:39.885 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T10:49:39.886 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:49:39.886 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:49:39.886 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local test_state=started 2023-10-11T10:49:39.886 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local status_result 2023-10-11T10:49:39.886 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_state=stopped 2023-10-11T10:49:39.887 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2023-10-11T10:49:39.887 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_instance=cluster1 2023-10-11T10:49:39.887 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ grep -i state 2023-10-11T10:49:39.887 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T10:49:39.887 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T10:49:39.888 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance=cluster1 2023-10-11T10:49:39.888 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local loop_instance 2023-10-11T10:49:39.888 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-10-11T10:49:39.888 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sleep 0 2023-10-11T10:49:39.891 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-10-11T10:49:39.891 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ seq 0 3 2023-10-11T10:49:39.891 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-10-11T10:49:39.891 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-10-11T10:49:39.891 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1:0 2023-10-11T10:49:39.892 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T10:49:39.892 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance 2023-10-11T10:49:39.892 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-10-11T10:49:39.892 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local val=cluster1:0 2023-10-11T10:49:39.893 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_var_name=cluster 2023-10-11T10:49:39.893 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance_var_name=instance 2023-10-11T10:49:39.893 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ cluster=cluster1 2023-10-11T10:49:39.893 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ instance=0 2023-10-11T10:49:39.893 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' 0 = cluster1:0 ']' 2023-10-11T10:49:39.894 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval cluster=cluster1 2023-10-11T10:49:39.894 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T10:49:39.894 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval instance=0 2023-10-11T10:49:39.894 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=0 2023-10-11T10:49:39.894 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-10-11T10:49:39.895 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local local_cluster=cluster1:0 2023-10-11T10:49:39.895 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster=cluster1 2023-10-11T10:49:39.895 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance 2023-10-11T10:49:39.895 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-10-11T10:49:39.895 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local val=cluster1:0 2023-10-11T10:49:39.895 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster_var_name=local_cluster 2023-10-11T10:49:39.896 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance_var_name=instance 2023-10-11T10:49:39.896 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T10:49:39.896 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=0 2023-10-11T10:49:39.896 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ '[' 0 = cluster1:0 ']' 2023-10-11T10:49:39.896 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval local_cluster=cluster1 2023-10-11T10:49:39.896 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ local_cluster=cluster1 2023-10-11T10:49:39.897 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval instance=0 2023-10-11T10:49:39.897 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ instance=0 2023-10-11T10:49:39.897 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-10-11T10:49:39.911 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T10:49:39.911 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T10:49:39.911 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T10:49:39.912 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-10-11T10:49:40.012 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-10-11T10:49:40.012 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd mirror status : get status for rbd mirror 2023-10-11T10:49:40.013 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:admin_socket: invalid command 2023-10-11T10:49:40.016 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-10-11T10:49:40.016 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-10-11T10:49:40.016 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1:1 2023-10-11T10:49:40.016 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T10:49:40.016 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance 2023-10-11T10:49:40.017 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-10-11T10:49:40.017 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local val=cluster1:1 2023-10-11T10:49:40.017 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_var_name=cluster 2023-10-11T10:49:40.017 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance_var_name=instance 2023-10-11T10:49:40.017 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ cluster=cluster1 2023-10-11T10:49:40.018 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ instance=1 2023-10-11T10:49:40.018 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' 1 = cluster1:1 ']' 2023-10-11T10:49:40.018 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval cluster=cluster1 2023-10-11T10:49:40.018 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T10:49:40.018 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval instance=1 2023-10-11T10:49:40.019 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=1 2023-10-11T10:49:40.019 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-10-11T10:49:40.019 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local local_cluster=cluster1:1 2023-10-11T10:49:40.019 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster=cluster1 2023-10-11T10:49:40.019 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance 2023-10-11T10:49:40.020 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-10-11T10:49:40.020 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local val=cluster1:1 2023-10-11T10:49:40.020 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster_var_name=local_cluster 2023-10-11T10:49:40.020 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance_var_name=instance 2023-10-11T10:49:40.020 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T10:49:40.020 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=1 2023-10-11T10:49:40.021 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ '[' 1 = cluster1:1 ']' 2023-10-11T10:49:40.021 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval local_cluster=cluster1 2023-10-11T10:49:40.021 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ local_cluster=cluster1 2023-10-11T10:49:40.021 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval instance=1 2023-10-11T10:49:40.021 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ instance=1 2023-10-11T10:49:40.021 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-10-11T10:49:40.034 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T10:49:40.035 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T10:49:40.035 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T10:49:40.035 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-10-11T10:49:40.140 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ return 0 2023-10-11T10:49:40.141 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ status_result=' "state": "Replaying"' 2023-10-11T10:49:40.141 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ echo ' "state": "Replaying"' 2023-10-11T10:49:40.141 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ grep -i Replaying 2023-10-11T10:49:40.142 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: "state": "Replaying" 2023-10-11T10:49:40.143 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ current_state=started 2023-10-11T10:49:40.144 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test started = started 2023-10-11T10:49:40.144 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T10:49:40.144 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2023-10-11T10:49:40.144 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local local_cluster=cluster1 2023-10-11T10:49:40.144 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:49:40.145 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:49:40.145 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:49:40.145 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' snapshot = journal ']' 2023-10-11T10:49:40.145 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' snapshot = snapshot ']' 2023-10-11T10:49:40.145 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2023-10-11T10:49:40.145 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local local_cluster=cluster1 2023-10-11T10:49:40.146 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:49:40.146 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:49:40.146 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:49:40.146 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ mkfname cluster2-mirror-test.status 2023-10-11T10:49:40.146 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ echo cluster2-mirror-test.status 2023-10-11T10:49:40.146 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -e 's|[/ ]|_|g' 2023-10-11T10:49:40.147 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local status_log=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.status 2023-10-11T10:49:40.149 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ mkfname cluster1-mirror-test.status 2023-10-11T10:49:40.150 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ echo cluster1-mirror-test.status 2023-10-11T10:49:40.150 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -e 's|[/ ]|_|g' 2023-10-11T10:49:40.150 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local local_status_log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.status 2023-10-11T10:49:40.151 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ mirror_image_snapshot cluster2 mirror test 2023-10-11T10:49:40.151 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:49:40.151 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:49:40.151 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:49:40.151 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2023-10-11T10:49:41.877 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:Snapshot ID: 25 2023-10-11T10:49:41.892 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.status 2023-10-11T10:49:41.893 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:49:41.893 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:49:41.893 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:49:41.893 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.status 2023-10-11T10:49:41.893 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2023-10-11T10:49:41.894 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T10:49:41.961 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-10-11T10:49:41.963 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snapshot_id=25 2023-10-11T10:49:41.963 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T10:49:41.963 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T10:49:41.964 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.2 2023-10-11T10:49:42.165 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.status 2023-10-11T10:49:42.165 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T10:49:42.165 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:49:42.165 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:49:42.165 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.status 2023-10-11T10:49:42.166 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-10-11T10:49:42.166 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T10:49:42.213 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T10:49:42.214 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T10:49:42.215 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:-:1.1: Document is empty 2023-10-11T10:49:42.215 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: 2023-10-11T10:49:42.215 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:^ 2023-10-11T10:49:42.216 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id= 2023-10-11T10:49:42.216 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 25 = '' 2023-10-11T10:49:42.216 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T10:49:42.217 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.4 2023-10-11T10:49:42.617 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.status 2023-10-11T10:49:42.617 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T10:49:42.617 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:49:42.618 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:49:42.618 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.status 2023-10-11T10:49:42.618 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-10-11T10:49:42.618 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T10:49:42.669 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T10:49:42.670 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T10:49:42.671 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:-:1.1: Document is empty 2023-10-11T10:49:42.671 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: 2023-10-11T10:49:42.671 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:^ 2023-10-11T10:49:42.672 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id= 2023-10-11T10:49:42.672 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 25 = '' 2023-10-11T10:49:42.672 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T10:49:42.672 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.8 2023-10-11T10:49:43.473 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.status 2023-10-11T10:49:43.473 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T10:49:43.474 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:49:43.474 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:49:43.474 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.status 2023-10-11T10:49:43.474 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-10-11T10:49:43.474 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T10:49:43.526 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T10:49:43.527 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T10:49:43.528 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:-:1.1: Document is empty 2023-10-11T10:49:43.529 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: 2023-10-11T10:49:43.529 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:^ 2023-10-11T10:49:43.529 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id= 2023-10-11T10:49:43.530 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 25 = '' 2023-10-11T10:49:43.530 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T10:49:43.530 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 1.6 2023-10-11T10:49:45.131 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.status 2023-10-11T10:49:45.131 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T10:49:45.132 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:49:45.132 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:49:45.132 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.status 2023-10-11T10:49:45.132 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-10-11T10:49:45.132 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T10:49:45.179 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T10:49:45.181 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id=25 2023-10-11T10:49:45.182 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 25 = 25 2023-10-11T10:49:45.182 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T10:49:45.182 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_snap_present cluster1 mirror test snap6 2023-10-11T10:49:45.182 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T10:49:45.182 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:49:45.183 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:49:45.183 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap6 2023-10-11T10:49:45.183 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T10:49:45.183 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-10-11T10:49:45.183 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 1 2023-10-11T10:49:46.182 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 info mirror/test@snap6 2023-10-11T10:49:46.233 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:rbd image 'test': 2023-10-11T10:49:46.233 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: size 512 MiB in 128 objects 2023-10-11T10:49:46.233 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: order 22 (4 MiB objects) 2023-10-11T10:49:46.233 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: snapshot_count: 12 2023-10-11T10:49:46.234 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: id: 10afefb6b972 2023-10-11T10:49:46.234 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: block_name_prefix: rbd_data.10afefb6b972 2023-10-11T10:49:46.234 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: format: 2 2023-10-11T10:49:46.234 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-10-11T10:49:46.234 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: op_features: 2023-10-11T10:49:46.234 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: flags: 2023-10-11T10:49:46.235 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: create_timestamp: Wed Oct 11 10:44:53 2023 2023-10-11T10:49:46.235 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: access_timestamp: Wed Oct 11 10:48:04 2023 2023-10-11T10:49:46.235 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: modify_timestamp: Wed Oct 11 10:44:53 2023 2023-10-11T10:49:46.235 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: protected: False 2023-10-11T10:49:46.235 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring state: enabled 2023-10-11T10:49:46.236 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring mode: snapshot 2023-10-11T10:49:46.236 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring global id: 3dd93db4-c9cc-462c-8b49-007eba9a90d2 2023-10-11T10:49:46.236 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring primary: false 2023-10-11T10:49:46.237 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T10:49:46.237 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n snap5-clean ']' 2023-10-11T10:49:46.237 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ compare_image_snaps mirror test snap5-clean 2023-10-11T10:49:46.238 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:49:46.238 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:49:46.238 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap5-clean 2023-10-11T10:49:46.238 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local ret=0 2023-10-11T10:49:46.238 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local rmt_export=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export 2023-10-11T10:49:46.239 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local loc_export=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:49:46.239 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rm -f /tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:49:46.239 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 -p mirror export test@snap5-clean /tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export 2023-10-11T10:49:46.815 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-10-11T10:49:46.822 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 -p mirror export test@snap5-clean /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:49:47.590 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-10-11T10:49:47.600 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ cmp /tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:49:47.814 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rm -f /tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:49:47.955 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T10:49:47.955 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ compare_image_snaps mirror test snap6 2023-10-11T10:49:47.956 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:49:47.956 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:49:47.956 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap6 2023-10-11T10:49:47.956 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local ret=0 2023-10-11T10:49:47.956 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local rmt_export=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export 2023-10-11T10:49:47.956 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local loc_export=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:49:47.957 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rm -f /tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:49:47.957 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 -p mirror export test@snap6 /tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export 2023-10-11T10:49:48.504 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.4827320083470108) > weight (0.25) 2023-10-11T10:49:48.504 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.9312917720174415) > weight (0.25) 2023-10-11T10:49:48.505 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.8149820597107222) > weight (0.25) 2023-10-11T10:49:48.505 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.3 2023-10-11T10:49:48.505 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Sent signal 15 2023-10-11T10:49:48.506 INFO:tasks.rbd_mirror_thrash:waiting for 5 secs before reviving daemons 2023-10-11T10:49:48.506 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi111.stderr:2023-10-11T10:49:48.506+0000 7fa1232d3700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 (PID: 106623) UID: 1000 2023-10-11T10:49:48.541 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-10-11T10:49:48.551 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 -p mirror export test@snap6 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:49:49.292 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-10-11T10:49:49.304 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ cmp /tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:49:49.516 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rm -f /tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:49:49.658 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T10:49:49.659 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ clean_snap_name=snap6-clean 2023-10-11T10:49:49.659 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ create_snap cluster2 mirror test snap6-clean 2023-10-11T10:49:49.659 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:49:49.659 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:49:49.659 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:49:49.660 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap6-clean 2023-10-11T10:49:49.660 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap6-clean --debug-rbd=20 --debug-journaler=20 2023-10-11T10:49:49.852 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 10` 2023-10-11T10:49:49.852 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ stress_write_image cluster2 mirror test 2023-10-11T10:49:49.853 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:49:49.853 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:49:49.853 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:49:49.853 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-10-11T10:49:49.854 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local duration=12 2023-10-11T10:49:49.854 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ set +e 2023-10-11T10:49:49.854 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2023-10-11T10:49:50.164 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:100 IOs, 64323584 bytes 2023-10-11T10:49:50.367 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:200 IOs, 99922944 bytes 2023-10-11T10:49:50.504 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:300 IOs, 151504896 bytes 2023-10-11T10:49:50.762 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:400 IOs, 221874176 bytes 2023-10-11T10:49:51.017 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:500 IOs, 319206400 bytes 2023-10-11T10:49:51.124 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:600 IOs, 336885760 bytes 2023-10-11T10:49:51.212 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:700 IOs, 346340352 bytes 2023-10-11T10:49:51.337 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:800 IOs, 379361280 bytes 2023-10-11T10:49:51.513 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:900 IOs, 448928768 bytes 2023-10-11T10:49:51.685 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1000 IOs, 471449600 bytes 2023-10-11T10:49:51.753 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1100 IOs, 472039424 bytes 2023-10-11T10:49:51.878 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1200 IOs, 509302784 bytes 2023-10-11T10:49:52.005 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1300 IOs, 529987584 bytes 2023-10-11T10:49:52.228 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1400 IOs, 608736256 bytes 2023-10-11T10:49:52.387 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1500 IOs, 650789888 bytes 2023-10-11T10:49:52.541 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1600 IOs, 688671744 bytes 2023-10-11T10:49:52.669 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1700 IOs, 721499136 bytes 2023-10-11T10:49:52.769 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1800 IOs, 729443328 bytes 2023-10-11T10:49:52.957 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1900 IOs, 784983040 bytes 2023-10-11T10:49:53.126 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2000 IOs, 822118400 bytes 2023-10-11T10:49:53.278 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2100 IOs, 864278528 bytes 2023-10-11T10:49:53.389 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2200 IOs, 886751232 bytes 2023-10-11T10:49:53.454 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2300 IOs, 887372800 bytes 2023-10-11T10:49:53.507 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.3 2023-10-11T10:49:53.507 INFO:teuthology.orchestra.run:waiting for 600 2023-10-11T10:49:53.508 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.3 2023-10-11T10:49:53.508 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Restarting daemon 2023-10-11T10:49:53.508 DEBUG:teuthology.orchestra.run.smithi111:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 2023-10-11T10:49:53.514 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Started 2023-10-11T10:49:53.515 INFO:tasks.rbd_mirror_thrash:waiting for 102 secs before thrashing 2023-10-11T10:49:53.610 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2400 IOs, 914339840 bytes 2023-10-11T10:49:53.740 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2500 IOs, 935587840 bytes 2023-10-11T10:49:53.925 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2600 IOs, 996151296 bytes 2023-10-11T10:49:54.086 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2700 IOs, 1041510400 bytes 2023-10-11T10:49:54.291 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2800 IOs, 1105143808 bytes 2023-10-11T10:49:54.487 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2900 IOs, 1167325184 bytes 2023-10-11T10:49:54.670 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3000 IOs, 1213917184 bytes 2023-10-11T10:49:54.798 INFO:tasks.ceph.cluster2.osd.2.smithi111.stderr:2023-10-11T10:49:54.796+0000 7fa35bd9b700 -1 reset not still connected to 0x5563f9787ad0 2023-10-11T10:49:54.814 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3100 IOs, 1299700736 bytes 2023-10-11T10:49:54.985 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3200 IOs, 1300342784 bytes 2023-10-11T10:49:55.114 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3300 IOs, 1322906624 bytes 2023-10-11T10:49:55.232 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3400 IOs, 1367399424 bytes 2023-10-11T10:49:55.703 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3500 IOs, 1483062272 bytes 2023-10-11T10:49:55.900 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3600 IOs, 1515545600 bytes 2023-10-11T10:49:56.031 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3700 IOs, 1547287552 bytes 2023-10-11T10:49:56.116 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3800 IOs, 1553195008 bytes 2023-10-11T10:49:56.271 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3900 IOs, 1591868416 bytes 2023-10-11T10:49:56.399 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4000 IOs, 1615825920 bytes 2023-10-11T10:49:56.518 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4100 IOs, 1639454720 bytes 2023-10-11T10:49:56.712 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4200 IOs, 1697400832 bytes 2023-10-11T10:49:56.818 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4300 IOs, 1713683456 bytes 2023-10-11T10:49:56.982 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4400 IOs, 1758422016 bytes 2023-10-11T10:49:57.246 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4500 IOs, 1850458112 bytes 2023-10-11T10:49:57.414 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4600 IOs, 1898439680 bytes 2023-10-11T10:49:57.548 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4700 IOs, 1922786304 bytes 2023-10-11T10:49:57.621 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4800 IOs, 1926138880 bytes 2023-10-11T10:49:57.749 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4900 IOs, 1957786624 bytes 2023-10-11T10:49:57.854 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5000 IOs, 1980835840 bytes 2023-10-11T10:49:57.973 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5100 IOs, 1995119616 bytes 2023-10-11T10:49:58.121 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5200 IOs, 2027537408 bytes 2023-10-11T10:49:58.339 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5300 IOs, 2098093056 bytes 2023-10-11T10:49:58.430 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5400 IOs, 2110836736 bytes 2023-10-11T10:49:58.552 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5500 IOs, 2132188160 bytes 2023-10-11T10:49:58.725 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5600 IOs, 2184946688 bytes 2023-10-11T10:49:58.955 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5700 IOs, 2242926592 bytes 2023-10-11T10:49:59.028 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5800 IOs, 2246268928 bytes 2023-10-11T10:49:59.189 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5900 IOs, 2268922880 bytes 2023-10-11T10:49:59.295 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6000 IOs, 2294603776 bytes 2023-10-11T10:49:59.474 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6100 IOs, 2334499840 bytes 2023-10-11T10:49:59.677 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6200 IOs, 2411356160 bytes 2023-10-11T10:49:59.843 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6300 IOs, 2442214400 bytes 2023-10-11T10:49:59.963 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6400 IOs, 2466129920 bytes 2023-10-11T10:50:00.194 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6500 IOs, 2545171456 bytes 2023-10-11T10:50:00.277 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6600 IOs, 2551766016 bytes 2023-10-11T10:50:00.496 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6700 IOs, 2622096384 bytes 2023-10-11T10:50:00.615 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6800 IOs, 2658864128 bytes 2023-10-11T10:50:00.844 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6900 IOs, 2724691968 bytes 2023-10-11T10:50:01.010 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:7000 IOs, 2786010112 bytes 2023-10-11T10:50:01.111 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:7100 IOs, 2786607104 bytes 2023-10-11T10:50:01.212 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:7200 IOs, 2798830592 bytes 2023-10-11T10:50:01.287 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:7300 IOs, 2799462400 bytes 2023-10-11T10:50:01.432 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:7400 IOs, 2836931584 bytes 2023-10-11T10:50:01.550 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:7500 IOs, 2856848384 bytes 2023-10-11T10:50:01.688 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:7600 IOs, 2885822464 bytes 2023-10-11T10:50:01.763 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:7700 IOs, 2886419456 bytes 2023-10-11T10:50:01.859 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ error_code=124 2023-10-11T10:50:01.860 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ set -e 2023-10-11T10:50:01.860 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' 124 -eq 124 ']' 2023-10-11T10:50:01.860 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T10:50:01.861 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2023-10-11T10:50:01.861 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T10:50:01.861 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:50:01.862 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:50:01.862 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local state_pattern=up+replaying 2023-10-11T10:50:01.863 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local description_pattern= 2023-10-11T10:50:01.863 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local service_pattern= 2023-10-11T10:50:01.863 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-10-11T10:50:01.864 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 1 2023-10-11T10:50:02.861 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2023-10-11T10:50:02.861 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T10:50:02.861 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:50:02.861 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:50:02.862 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local state_pattern=up+replaying 2023-10-11T10:50:02.862 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local description_pattern= 2023-10-11T10:50:02.862 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local service_pattern= 2023-10-11T10:50:02.862 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ mkfname cluster1-mirror-test.mirror_status 2023-10-11T10:50:02.862 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ echo cluster1-mirror-test.mirror_status 2023-10-11T10:50:02.863 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -e 's|[/ ]|_|g' 2023-10-11T10:50:02.863 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local status_log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.mirror_status 2023-10-11T10:50:02.864 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ CEPH_ARGS= 2023-10-11T10:50:02.864 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2023-10-11T10:50:02.864 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ tee /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.mirror_status 2023-10-11T10:50:02.916 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:test: 2023-10-11T10:50:02.916 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: global_id: 3dd93db4-c9cc-462c-8b49-007eba9a90d2 2023-10-11T10:50:02.917 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: state: up+replaying 2023-10-11T10:50:02.917 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: description: replaying, {"bytes_per_second":17759914.67,"bytes_per_snapshot":527706624.0,"last_snapshot_bytes":532797440,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1697021380,"remote_snapshot_timestamp":1697021380,"replay_state":"idle"} 2023-10-11T10:50:02.917 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: service: mirror.1 on smithi111 2023-10-11T10:50:02.917 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: last_update: 2023-10-11 10:49:51 2023-10-11T10:50:02.917 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: peer_sites: 2023-10-11T10:50:02.918 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: name: cluster2 2023-10-11T10:50:02.918 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: state: up+stopped 2023-10-11T10:50:02.918 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: description: local image is primary 2023-10-11T10:50:02.918 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: last_update: 2023-10-11 10:49:51 2023-10-11T10:50:02.921 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.mirror_status 2023-10-11T10:50:02.922 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: state: up+replaying 2023-10-11T10:50:02.922 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ grep '^ description: .*' /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.mirror_status 2023-10-11T10:50:02.923 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: description: replaying, {"bytes_per_second":17759914.67,"bytes_per_snapshot":527706624.0,"last_snapshot_bytes":532797440,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1697021380,"remote_snapshot_timestamp":1697021380,"replay_state":"idle"} 2023-10-11T10:50:02.924 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n '' ']' 2023-10-11T10:50:02.924 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ echo up+replaying 2023-10-11T10:50:02.924 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ grep '^up+' 2023-10-11T10:50:02.925 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:up+replaying 2023-10-11T10:50:02.925 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ grep 'service: *mirror..* on ' /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.mirror_status 2023-10-11T10:50:02.925 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: service: mirror.1 on smithi111 2023-10-11T10:50:02.926 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.mirror_status 2023-10-11T10:50:02.927 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local 'last_update=2023-10-11 10:49:51' 2023-10-11T10:50:02.927 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2023-10-11 10:49:51' 2023-10-11T10:50:02.928 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T10:50:02.928 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:50:02.928 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:50:02.928 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local state_pattern=up+replaying 2023-10-11T10:50:02.928 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local 'prev_last_update=2023-10-11 10:49:51' 2023-10-11T10:50:02.928 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ mkfname cluster1-mirror.mirror_status 2023-10-11T10:50:02.929 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ echo cluster1-mirror.mirror_status 2023-10-11T10:50:02.929 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -e 's|[/ ]|_|g' 2023-10-11T10:50:02.929 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local status_log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror.mirror_status 2023-10-11T10:50:02.930 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2023-10-11T10:50:02.965 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd: failed to query services: (13) Permission denied 2023-10-11T10:50:02.991 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local last_update state 2023-10-11T10:50:02.991 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2023-10-11T10:50:02.993 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ last_update='2023-10-11 10:49:51' 2023-10-11T10:50:02.993 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2023-10-11T10:50:02.995 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ state=up+replaying 2023-10-11T10:50:02.995 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ echo up+replaying 2023-10-11T10:50:02.995 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ grep up+replaying 2023-10-11T10:50:02.996 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:up+replaying 2023-10-11T10:50:02.997 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T10:50:02.997 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T10:50:02.997 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ snap_name=snap7 2023-10-11T10:50:02.997 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ create_snap cluster2 mirror test snap7 2023-10-11T10:50:02.997 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:50:02.998 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:50:02.998 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:50:02.998 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap7 2023-10-11T10:50:02.998 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap7 --debug-rbd=20 --debug-journaler=20 2023-10-11T10:50:31.907 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_replay_started cluster1 mirror test 2023-10-11T10:50:31.907 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T10:50:31.907 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:50:31.907 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:50:31.908 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2023-10-11T10:50:31.908 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T10:50:31.908 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:50:31.908 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:50:31.908 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local state=started 2023-10-11T10:50:31.908 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T10:50:31.909 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-10-11T10:50:31.909 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 1 2023-10-11T10:50:32.907 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test_image_replay_state cluster1 mirror test started 2023-10-11T10:50:32.908 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T10:50:32.908 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:50:32.908 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:50:32.908 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local test_state=started 2023-10-11T10:50:32.909 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local status_result 2023-10-11T10:50:32.909 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_state=stopped 2023-10-11T10:50:32.910 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2023-10-11T10:50:32.910 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_instance=cluster1 2023-10-11T10:50:32.910 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T10:50:32.910 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ grep -i state 2023-10-11T10:50:32.911 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T10:50:32.911 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance=cluster1 2023-10-11T10:50:32.911 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local loop_instance 2023-10-11T10:50:32.911 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-10-11T10:50:32.911 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sleep 0 2023-10-11T10:50:32.911 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-10-11T10:50:32.912 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ seq 0 3 2023-10-11T10:50:32.917 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-10-11T10:50:32.917 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-10-11T10:50:32.918 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1:0 2023-10-11T10:50:32.918 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T10:50:32.919 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance 2023-10-11T10:50:32.919 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-10-11T10:50:32.919 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local val=cluster1:0 2023-10-11T10:50:32.920 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_var_name=cluster 2023-10-11T10:50:32.920 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance_var_name=instance 2023-10-11T10:50:32.920 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ cluster=cluster1 2023-10-11T10:50:32.920 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ instance=0 2023-10-11T10:50:32.921 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' 0 = cluster1:0 ']' 2023-10-11T10:50:32.921 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval cluster=cluster1 2023-10-11T10:50:32.921 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T10:50:32.921 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval instance=0 2023-10-11T10:50:32.921 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=0 2023-10-11T10:50:32.922 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-10-11T10:50:32.922 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local local_cluster=cluster1:0 2023-10-11T10:50:32.922 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster=cluster1 2023-10-11T10:50:32.922 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance 2023-10-11T10:50:32.922 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-10-11T10:50:32.922 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local val=cluster1:0 2023-10-11T10:50:32.923 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster_var_name=local_cluster 2023-10-11T10:50:32.923 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance_var_name=instance 2023-10-11T10:50:32.923 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T10:50:32.923 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=0 2023-10-11T10:50:32.923 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ '[' 0 = cluster1:0 ']' 2023-10-11T10:50:32.924 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval local_cluster=cluster1 2023-10-11T10:50:32.924 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ local_cluster=cluster1 2023-10-11T10:50:32.924 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval instance=0 2023-10-11T10:50:32.924 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ instance=0 2023-10-11T10:50:32.924 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-10-11T10:50:32.934 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T10:50:32.935 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T10:50:32.935 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T10:50:32.935 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-10-11T10:50:33.036 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-10-11T10:50:33.036 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd mirror status : get status for rbd mirror 2023-10-11T10:50:33.037 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:admin_socket: invalid command 2023-10-11T10:50:33.039 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-10-11T10:50:33.039 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-10-11T10:50:33.040 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1:1 2023-10-11T10:50:33.040 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T10:50:33.040 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance 2023-10-11T10:50:33.040 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-10-11T10:50:33.040 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local val=cluster1:1 2023-10-11T10:50:33.040 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_var_name=cluster 2023-10-11T10:50:33.041 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance_var_name=instance 2023-10-11T10:50:33.041 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ cluster=cluster1 2023-10-11T10:50:33.041 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ instance=1 2023-10-11T10:50:33.041 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' 1 = cluster1:1 ']' 2023-10-11T10:50:33.041 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval cluster=cluster1 2023-10-11T10:50:33.041 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T10:50:33.042 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval instance=1 2023-10-11T10:50:33.042 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=1 2023-10-11T10:50:33.042 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-10-11T10:50:33.042 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local local_cluster=cluster1:1 2023-10-11T10:50:33.042 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster=cluster1 2023-10-11T10:50:33.043 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance 2023-10-11T10:50:33.043 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-10-11T10:50:33.043 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local val=cluster1:1 2023-10-11T10:50:33.043 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster_var_name=local_cluster 2023-10-11T10:50:33.043 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance_var_name=instance 2023-10-11T10:50:33.043 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T10:50:33.044 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=1 2023-10-11T10:50:33.044 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ '[' 1 = cluster1:1 ']' 2023-10-11T10:50:33.044 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval local_cluster=cluster1 2023-10-11T10:50:33.044 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ local_cluster=cluster1 2023-10-11T10:50:33.044 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval instance=1 2023-10-11T10:50:33.044 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ instance=1 2023-10-11T10:50:33.045 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-10-11T10:50:33.058 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T10:50:33.058 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T10:50:33.058 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T10:50:33.058 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-10-11T10:50:33.162 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ return 0 2023-10-11T10:50:33.162 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ status_result=' "state": "Replaying"' 2023-10-11T10:50:33.162 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ echo ' "state": "Replaying"' 2023-10-11T10:50:33.163 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ grep -i Replaying 2023-10-11T10:50:33.163 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: "state": "Replaying" 2023-10-11T10:50:33.165 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ current_state=started 2023-10-11T10:50:33.165 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test started = started 2023-10-11T10:50:33.165 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T10:50:33.165 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2023-10-11T10:50:33.165 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local local_cluster=cluster1 2023-10-11T10:50:33.166 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:50:33.166 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:50:33.166 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:50:33.166 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' snapshot = journal ']' 2023-10-11T10:50:33.167 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' snapshot = snapshot ']' 2023-10-11T10:50:33.167 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2023-10-11T10:50:33.167 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local local_cluster=cluster1 2023-10-11T10:50:33.167 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:50:33.168 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:50:33.168 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:50:33.168 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ mkfname cluster2-mirror-test.status 2023-10-11T10:50:33.169 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ echo cluster2-mirror-test.status 2023-10-11T10:50:33.169 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -e 's|[/ ]|_|g' 2023-10-11T10:50:33.171 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local status_log=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.status 2023-10-11T10:50:33.171 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ mkfname cluster1-mirror-test.status 2023-10-11T10:50:33.172 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ echo cluster1-mirror-test.status 2023-10-11T10:50:33.172 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -e 's|[/ ]|_|g' 2023-10-11T10:50:33.172 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local local_status_log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.status 2023-10-11T10:50:33.172 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ mirror_image_snapshot cluster2 mirror test 2023-10-11T10:50:33.172 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:50:33.173 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:50:33.173 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:50:33.173 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2023-10-11T10:50:34.929 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:Snapshot ID: 29 2023-10-11T10:50:34.948 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.status 2023-10-11T10:50:34.948 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:50:34.948 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:50:34.948 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:50:34.948 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.status 2023-10-11T10:50:34.949 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2023-10-11T10:50:34.950 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T10:50:35.012 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-10-11T10:50:35.015 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snapshot_id=29 2023-10-11T10:50:35.015 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T10:50:35.015 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T10:50:35.015 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.2 2023-10-11T10:50:35.216 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.status 2023-10-11T10:50:35.217 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T10:50:35.217 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:50:35.217 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:50:35.217 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.status 2023-10-11T10:50:35.217 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-10-11T10:50:35.217 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T10:50:35.263 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T10:50:35.264 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T10:50:35.265 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:-:1.1: Document is empty 2023-10-11T10:50:35.265 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: 2023-10-11T10:50:35.265 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:^ 2023-10-11T10:50:35.266 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id= 2023-10-11T10:50:35.266 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 29 = '' 2023-10-11T10:50:35.266 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T10:50:35.266 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.4 2023-10-11T10:50:35.666 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.status 2023-10-11T10:50:35.667 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T10:50:35.667 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:50:35.667 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:50:35.667 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.status 2023-10-11T10:50:35.667 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-10-11T10:50:35.668 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T10:50:35.714 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T10:50:35.715 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T10:50:35.716 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:-:1.1: Document is empty 2023-10-11T10:50:35.716 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: 2023-10-11T10:50:35.716 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:^ 2023-10-11T10:50:35.717 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id= 2023-10-11T10:50:35.717 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 29 = '' 2023-10-11T10:50:35.717 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T10:50:35.717 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.8 2023-10-11T10:50:36.518 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.status 2023-10-11T10:50:36.518 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T10:50:36.518 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:50:36.519 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:50:36.519 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.status 2023-10-11T10:50:36.519 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-10-11T10:50:36.519 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T10:50:36.572 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T10:50:36.573 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T10:50:36.574 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:-:1.1: Document is empty 2023-10-11T10:50:36.574 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: 2023-10-11T10:50:36.574 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:^ 2023-10-11T10:50:36.575 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id= 2023-10-11T10:50:36.575 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 29 = '' 2023-10-11T10:50:36.575 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T10:50:36.576 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 1.6 2023-10-11T10:50:38.176 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.status 2023-10-11T10:50:38.176 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T10:50:38.177 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:50:38.177 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:50:38.177 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.status 2023-10-11T10:50:38.177 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-10-11T10:50:38.177 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T10:50:38.224 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T10:50:38.227 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id=29 2023-10-11T10:50:38.227 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 29 = 29 2023-10-11T10:50:38.227 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T10:50:38.227 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_snap_present cluster1 mirror test snap7 2023-10-11T10:50:38.227 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T10:50:38.228 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:50:38.228 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:50:38.228 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap7 2023-10-11T10:50:38.228 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T10:50:38.228 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-10-11T10:50:38.229 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 1 2023-10-11T10:50:39.227 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 info mirror/test@snap7 2023-10-11T10:50:39.275 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:rbd image 'test': 2023-10-11T10:50:39.275 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: size 512 MiB in 128 objects 2023-10-11T10:50:39.276 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: order 22 (4 MiB objects) 2023-10-11T10:50:39.276 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: snapshot_count: 14 2023-10-11T10:50:39.276 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: id: 10afefb6b972 2023-10-11T10:50:39.276 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: block_name_prefix: rbd_data.10afefb6b972 2023-10-11T10:50:39.277 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: format: 2 2023-10-11T10:50:39.277 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-10-11T10:50:39.277 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: op_features: 2023-10-11T10:50:39.277 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: flags: 2023-10-11T10:50:39.277 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: create_timestamp: Wed Oct 11 10:44:53 2023 2023-10-11T10:50:39.277 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: access_timestamp: Wed Oct 11 10:49:46 2023 2023-10-11T10:50:39.278 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: modify_timestamp: Wed Oct 11 10:44:53 2023 2023-10-11T10:50:39.278 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: protected: False 2023-10-11T10:50:39.278 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring state: enabled 2023-10-11T10:50:39.278 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring mode: snapshot 2023-10-11T10:50:39.278 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring global id: 3dd93db4-c9cc-462c-8b49-007eba9a90d2 2023-10-11T10:50:39.279 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring primary: false 2023-10-11T10:50:39.280 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T10:50:39.280 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n snap6-clean ']' 2023-10-11T10:50:39.280 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ compare_image_snaps mirror test snap6-clean 2023-10-11T10:50:39.281 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:50:39.281 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:50:39.281 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap6-clean 2023-10-11T10:50:39.281 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local ret=0 2023-10-11T10:50:39.281 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local rmt_export=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export 2023-10-11T10:50:39.281 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local loc_export=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:50:39.282 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rm -f /tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:50:39.282 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 -p mirror export test@snap6-clean /tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export 2023-10-11T10:50:39.869 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-10-11T10:50:39.879 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 -p mirror export test@snap6-clean /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:50:40.573 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-10-11T10:50:40.580 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ cmp /tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:50:40.795 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rm -f /tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:50:40.936 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T10:50:40.936 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ compare_image_snaps mirror test snap7 2023-10-11T10:50:40.936 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:50:40.936 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:50:40.937 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap7 2023-10-11T10:50:40.937 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local ret=0 2023-10-11T10:50:40.937 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local rmt_export=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export 2023-10-11T10:50:40.937 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local loc_export=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:50:40.937 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rm -f /tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:50:40.938 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 -p mirror export test@snap7 /tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export 2023-10-11T10:50:41.520 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-10-11T10:50:41.528 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 -p mirror export test@snap7 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:50:42.206 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-10-11T10:50:42.216 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ cmp /tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:50:42.428 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rm -f /tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:50:42.566 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T10:50:42.567 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ clean_snap_name=snap7-clean 2023-10-11T10:50:42.567 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ create_snap cluster2 mirror test snap7-clean 2023-10-11T10:50:42.567 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:50:42.567 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:50:42.567 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:50:42.567 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap7-clean 2023-10-11T10:50:42.568 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap7-clean --debug-rbd=20 --debug-journaler=20 2023-10-11T10:50:42.863 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 10` 2023-10-11T10:50:42.864 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ stress_write_image cluster2 mirror test 2023-10-11T10:50:42.864 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:50:42.864 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:50:42.864 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:50:42.864 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-10-11T10:50:42.865 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local duration=6 2023-10-11T10:50:42.865 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ set +e 2023-10-11T10:50:42.865 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2023-10-11T10:50:43.312 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:100 IOs, 39078912 bytes 2023-10-11T10:50:43.493 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:200 IOs, 73211904 bytes 2023-10-11T10:50:43.714 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:300 IOs, 154948608 bytes 2023-10-11T10:50:43.923 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:400 IOs, 215048192 bytes 2023-10-11T10:50:44.050 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:500 IOs, 245299200 bytes 2023-10-11T10:50:44.216 INFO:tasks.ceph.cluster2.osd.2.smithi111.stderr:2023-10-11T10:50:44.215+0000 7fa35bd9b700 -1 reset not still connected to 0x5563fb45de10 2023-10-11T10:50:44.265 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:600 IOs, 313741312 bytes 2023-10-11T10:50:44.377 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:700 IOs, 326625280 bytes 2023-10-11T10:50:44.455 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:800 IOs, 342848512 bytes 2023-10-11T10:50:44.515 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.46826965274504295) > weight (0.25) 2023-10-11T10:50:44.515 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.9503311097824497) > weight (0.25) 2023-10-11T10:50:44.515 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.5451802269465514) > weight (0.25) 2023-10-11T10:50:44.516 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.49147217877605753) > weight (0.25) 2023-10-11T10:50:44.516 INFO:tasks.rbd_mirror_thrash:waiting for 72 secs before thrashing 2023-10-11T10:50:44.646 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:900 IOs, 390549504 bytes 2023-10-11T10:50:44.801 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1000 IOs, 424047616 bytes 2023-10-11T10:50:45.021 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1100 IOs, 480883712 bytes 2023-10-11T10:50:45.122 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1200 IOs, 497161216 bytes 2023-10-11T10:50:45.388 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1300 IOs, 602508288 bytes 2023-10-11T10:50:45.534 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1400 IOs, 650085376 bytes 2023-10-11T10:50:45.753 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1500 IOs, 704187392 bytes 2023-10-11T10:50:45.943 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1600 IOs, 769448960 bytes 2023-10-11T10:50:46.154 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1700 IOs, 841146368 bytes 2023-10-11T10:50:46.331 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1800 IOs, 907439104 bytes 2023-10-11T10:50:46.574 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1900 IOs, 996675584 bytes 2023-10-11T10:50:46.800 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2000 IOs, 1059211264 bytes 2023-10-11T10:50:46.973 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2100 IOs, 1106540544 bytes 2023-10-11T10:50:47.089 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2200 IOs, 1125308416 bytes 2023-10-11T10:50:47.265 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2300 IOs, 1174734848 bytes 2023-10-11T10:50:47.407 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2400 IOs, 1212658688 bytes 2023-10-11T10:50:47.572 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2500 IOs, 1263256576 bytes 2023-10-11T10:50:47.699 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2600 IOs, 1298910208 bytes 2023-10-11T10:50:47.918 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2700 IOs, 1359175680 bytes 2023-10-11T10:50:48.029 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2800 IOs, 1376237568 bytes 2023-10-11T10:50:48.233 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2900 IOs, 1446185984 bytes 2023-10-11T10:50:48.375 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3000 IOs, 1478586368 bytes 2023-10-11T10:50:48.466 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3100 IOs, 1481400320 bytes 2023-10-11T10:50:48.628 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3200 IOs, 1528335360 bytes 2023-10-11T10:50:48.870 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ error_code=124 2023-10-11T10:50:48.870 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ set -e 2023-10-11T10:50:48.870 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' 124 -eq 124 ']' 2023-10-11T10:50:48.871 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T10:50:48.871 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2023-10-11T10:50:48.871 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T10:50:48.871 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:50:48.871 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:50:48.872 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local state_pattern=up+replaying 2023-10-11T10:50:48.872 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local description_pattern= 2023-10-11T10:50:48.872 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local service_pattern= 2023-10-11T10:50:48.872 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-10-11T10:50:48.872 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 1 2023-10-11T10:50:49.872 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2023-10-11T10:50:49.873 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T10:50:49.873 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:50:49.873 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:50:49.873 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local state_pattern=up+replaying 2023-10-11T10:50:49.874 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local description_pattern= 2023-10-11T10:50:49.874 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local service_pattern= 2023-10-11T10:50:49.874 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ mkfname cluster1-mirror-test.mirror_status 2023-10-11T10:50:49.874 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ echo cluster1-mirror-test.mirror_status 2023-10-11T10:50:49.875 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -e 's|[/ ]|_|g' 2023-10-11T10:50:49.875 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local status_log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.mirror_status 2023-10-11T10:50:49.875 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ CEPH_ARGS= 2023-10-11T10:50:49.876 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2023-10-11T10:50:49.876 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ tee /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.mirror_status 2023-10-11T10:50:49.925 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:test: 2023-10-11T10:50:49.925 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: global_id: 3dd93db4-c9cc-462c-8b49-007eba9a90d2 2023-10-11T10:50:49.925 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: state: up+replaying 2023-10-11T10:50:49.926 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: description: replaying, {"bytes_per_second":17759914.67,"bytes_per_snapshot":527706624.0,"last_snapshot_bytes":532797440,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1697021380,"remote_snapshot_timestamp":1697021380,"replay_state":"idle"} 2023-10-11T10:50:49.926 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: service: mirror.1 on smithi111 2023-10-11T10:50:49.926 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: last_update: 2023-10-11 10:50:21 2023-10-11T10:50:49.926 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: peer_sites: 2023-10-11T10:50:49.926 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: name: cluster2 2023-10-11T10:50:49.927 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: state: up+stopped 2023-10-11T10:50:49.927 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: description: local image is primary 2023-10-11T10:50:49.927 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: last_update: 2023-10-11 10:50:21 2023-10-11T10:50:49.930 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.mirror_status 2023-10-11T10:50:49.932 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: state: up+replaying 2023-10-11T10:50:49.932 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ grep '^ description: .*' /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.mirror_status 2023-10-11T10:50:49.932 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: description: replaying, {"bytes_per_second":17759914.67,"bytes_per_snapshot":527706624.0,"last_snapshot_bytes":532797440,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1697021380,"remote_snapshot_timestamp":1697021380,"replay_state":"idle"} 2023-10-11T10:50:49.933 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n '' ']' 2023-10-11T10:50:49.933 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ echo up+replaying 2023-10-11T10:50:49.934 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ grep '^up+' 2023-10-11T10:50:49.934 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:up+replaying 2023-10-11T10:50:49.935 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ grep 'service: *mirror..* on ' /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.mirror_status 2023-10-11T10:50:49.935 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: service: mirror.1 on smithi111 2023-10-11T10:50:49.935 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.mirror_status 2023-10-11T10:50:49.937 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local 'last_update=2023-10-11 10:50:21' 2023-10-11T10:50:49.937 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2023-10-11 10:50:21' 2023-10-11T10:50:49.938 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T10:50:49.938 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:50:49.938 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:50:49.938 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local state_pattern=up+replaying 2023-10-11T10:50:49.938 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local 'prev_last_update=2023-10-11 10:50:21' 2023-10-11T10:50:49.939 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ mkfname cluster1-mirror.mirror_status 2023-10-11T10:50:49.939 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ echo cluster1-mirror.mirror_status 2023-10-11T10:50:49.939 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -e 's|[/ ]|_|g' 2023-10-11T10:50:49.940 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local status_log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror.mirror_status 2023-10-11T10:50:49.940 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2023-10-11T10:50:49.976 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd: failed to query services: (13) Permission denied 2023-10-11T10:50:49.999 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local last_update state 2023-10-11T10:50:50.000 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2023-10-11T10:50:50.001 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ last_update='2023-10-11 10:50:21' 2023-10-11T10:50:50.002 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2023-10-11T10:50:50.003 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ state=up+replaying 2023-10-11T10:50:50.004 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ echo up+replaying 2023-10-11T10:50:50.004 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ grep up+replaying 2023-10-11T10:50:50.004 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:up+replaying 2023-10-11T10:50:50.005 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T10:50:50.006 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T10:50:50.006 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ snap_name=snap8 2023-10-11T10:50:50.006 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ create_snap cluster2 mirror test snap8 2023-10-11T10:50:50.006 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:50:50.006 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:50:50.007 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:50:50.007 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap8 2023-10-11T10:50:50.007 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap8 --debug-rbd=20 --debug-journaler=20 2023-10-11T10:51:19.920 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_replay_started cluster1 mirror test 2023-10-11T10:51:19.921 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T10:51:19.921 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:51:19.921 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:51:19.921 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2023-10-11T10:51:19.921 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T10:51:19.922 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:51:19.922 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:51:19.922 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local state=started 2023-10-11T10:51:19.922 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T10:51:19.922 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-10-11T10:51:19.923 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 1 2023-10-11T10:51:20.921 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test_image_replay_state cluster1 mirror test started 2023-10-11T10:51:20.921 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T10:51:20.922 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:51:20.922 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:51:20.922 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local test_state=started 2023-10-11T10:51:20.922 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local status_result 2023-10-11T10:51:20.922 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_state=stopped 2023-10-11T10:51:20.923 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2023-10-11T10:51:20.923 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_instance=cluster1 2023-10-11T10:51:20.924 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T10:51:20.924 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ grep -i state 2023-10-11T10:51:20.924 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T10:51:20.924 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance=cluster1 2023-10-11T10:51:20.924 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local loop_instance 2023-10-11T10:51:20.925 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-10-11T10:51:20.925 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sleep 0 2023-10-11T10:51:20.925 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-10-11T10:51:20.925 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ seq 0 3 2023-10-11T10:51:20.927 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-10-11T10:51:20.928 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-10-11T10:51:20.928 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1:0 2023-10-11T10:51:20.928 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T10:51:20.928 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance 2023-10-11T10:51:20.928 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-10-11T10:51:20.929 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local val=cluster1:0 2023-10-11T10:51:20.929 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_var_name=cluster 2023-10-11T10:51:20.929 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance_var_name=instance 2023-10-11T10:51:20.929 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ cluster=cluster1 2023-10-11T10:51:20.929 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ instance=0 2023-10-11T10:51:20.929 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' 0 = cluster1:0 ']' 2023-10-11T10:51:20.930 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval cluster=cluster1 2023-10-11T10:51:20.930 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T10:51:20.930 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval instance=0 2023-10-11T10:51:20.930 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=0 2023-10-11T10:51:20.930 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-10-11T10:51:20.930 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local local_cluster=cluster1:0 2023-10-11T10:51:20.931 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster=cluster1 2023-10-11T10:51:20.931 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance 2023-10-11T10:51:20.931 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-10-11T10:51:20.931 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local val=cluster1:0 2023-10-11T10:51:20.931 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster_var_name=local_cluster 2023-10-11T10:51:20.931 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance_var_name=instance 2023-10-11T10:51:20.932 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T10:51:20.932 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=0 2023-10-11T10:51:20.932 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ '[' 0 = cluster1:0 ']' 2023-10-11T10:51:20.932 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval local_cluster=cluster1 2023-10-11T10:51:20.932 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ local_cluster=cluster1 2023-10-11T10:51:20.932 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval instance=0 2023-10-11T10:51:20.933 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ instance=0 2023-10-11T10:51:20.933 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-10-11T10:51:20.948 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T10:51:20.948 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T10:51:20.948 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T10:51:20.948 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-10-11T10:51:21.048 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-10-11T10:51:21.049 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd mirror status : get status for rbd mirror 2023-10-11T10:51:21.049 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:admin_socket: invalid command 2023-10-11T10:51:21.052 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-10-11T10:51:21.052 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-10-11T10:51:21.052 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1:1 2023-10-11T10:51:21.052 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T10:51:21.052 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance 2023-10-11T10:51:21.052 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-10-11T10:51:21.053 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local val=cluster1:1 2023-10-11T10:51:21.053 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_var_name=cluster 2023-10-11T10:51:21.053 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance_var_name=instance 2023-10-11T10:51:21.053 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ cluster=cluster1 2023-10-11T10:51:21.053 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ instance=1 2023-10-11T10:51:21.054 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' 1 = cluster1:1 ']' 2023-10-11T10:51:21.054 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval cluster=cluster1 2023-10-11T10:51:21.054 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T10:51:21.054 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval instance=1 2023-10-11T10:51:21.054 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=1 2023-10-11T10:51:21.054 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-10-11T10:51:21.055 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local local_cluster=cluster1:1 2023-10-11T10:51:21.055 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster=cluster1 2023-10-11T10:51:21.055 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance 2023-10-11T10:51:21.055 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-10-11T10:51:21.055 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local val=cluster1:1 2023-10-11T10:51:21.055 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster_var_name=local_cluster 2023-10-11T10:51:21.056 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance_var_name=instance 2023-10-11T10:51:21.056 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T10:51:21.056 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=1 2023-10-11T10:51:21.056 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ '[' 1 = cluster1:1 ']' 2023-10-11T10:51:21.056 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval local_cluster=cluster1 2023-10-11T10:51:21.057 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ local_cluster=cluster1 2023-10-11T10:51:21.057 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval instance=1 2023-10-11T10:51:21.057 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ instance=1 2023-10-11T10:51:21.057 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-10-11T10:51:21.070 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T10:51:21.070 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T10:51:21.071 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T10:51:21.071 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-10-11T10:51:21.175 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ return 0 2023-10-11T10:51:21.176 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ status_result=' "state": "Replaying"' 2023-10-11T10:51:21.176 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ echo ' "state": "Replaying"' 2023-10-11T10:51:21.176 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ grep -i Replaying 2023-10-11T10:51:21.177 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: "state": "Replaying" 2023-10-11T10:51:21.178 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ current_state=started 2023-10-11T10:51:21.178 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test started = started 2023-10-11T10:51:21.179 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T10:51:21.179 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2023-10-11T10:51:21.179 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local local_cluster=cluster1 2023-10-11T10:51:21.179 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:51:21.179 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:51:21.179 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:51:21.180 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' snapshot = journal ']' 2023-10-11T10:51:21.180 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' snapshot = snapshot ']' 2023-10-11T10:51:21.180 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2023-10-11T10:51:21.180 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local local_cluster=cluster1 2023-10-11T10:51:21.180 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:51:21.180 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:51:21.181 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:51:21.181 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ mkfname cluster2-mirror-test.status 2023-10-11T10:51:21.181 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ echo cluster2-mirror-test.status 2023-10-11T10:51:21.181 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -e 's|[/ ]|_|g' 2023-10-11T10:51:21.182 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local status_log=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.status 2023-10-11T10:51:21.183 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ mkfname cluster1-mirror-test.status 2023-10-11T10:51:21.183 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ echo cluster1-mirror-test.status 2023-10-11T10:51:21.183 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -e 's|[/ ]|_|g' 2023-10-11T10:51:21.183 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local local_status_log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.status 2023-10-11T10:51:21.183 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ mirror_image_snapshot cluster2 mirror test 2023-10-11T10:51:21.184 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:51:21.184 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:51:21.184 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:51:21.184 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2023-10-11T10:51:22.913 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:Snapshot ID: 33 2023-10-11T10:51:22.938 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.status 2023-10-11T10:51:22.939 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:51:22.939 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:51:22.939 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:51:22.939 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.status 2023-10-11T10:51:22.940 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T10:51:22.940 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2023-10-11T10:51:23.004 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-10-11T10:51:23.006 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snapshot_id=33 2023-10-11T10:51:23.006 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T10:51:23.007 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T10:51:23.007 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.2 2023-10-11T10:51:23.207 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.status 2023-10-11T10:51:23.207 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T10:51:23.208 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:51:23.208 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:51:23.208 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.status 2023-10-11T10:51:23.208 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-10-11T10:51:23.208 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T10:51:23.257 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T10:51:23.258 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T10:51:23.259 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:-:1.1: Document is empty 2023-10-11T10:51:23.259 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: 2023-10-11T10:51:23.259 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:^ 2023-10-11T10:51:23.260 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id= 2023-10-11T10:51:23.260 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 33 = '' 2023-10-11T10:51:23.260 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T10:51:23.260 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.4 2023-10-11T10:51:23.661 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.status 2023-10-11T10:51:23.661 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T10:51:23.661 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:51:23.662 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:51:23.662 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.status 2023-10-11T10:51:23.662 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-10-11T10:51:23.662 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T10:51:23.716 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T10:51:23.716 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T10:51:23.717 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:-:1.1: Document is empty 2023-10-11T10:51:23.717 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: 2023-10-11T10:51:23.717 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:^ 2023-10-11T10:51:23.718 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id= 2023-10-11T10:51:23.718 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 33 = '' 2023-10-11T10:51:23.719 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T10:51:23.719 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.8 2023-10-11T10:51:24.519 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.status 2023-10-11T10:51:24.519 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T10:51:24.520 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:51:24.520 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:51:24.520 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.status 2023-10-11T10:51:24.520 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-10-11T10:51:24.520 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T10:51:24.569 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T10:51:24.569 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T10:51:24.570 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:-:1.1: Document is empty 2023-10-11T10:51:24.571 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: 2023-10-11T10:51:24.571 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:^ 2023-10-11T10:51:24.571 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id= 2023-10-11T10:51:24.571 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 33 = '' 2023-10-11T10:51:24.571 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T10:51:24.572 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 1.6 2023-10-11T10:51:26.172 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.status 2023-10-11T10:51:26.172 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T10:51:26.173 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:51:26.173 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:51:26.173 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.status 2023-10-11T10:51:26.173 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-10-11T10:51:26.173 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T10:51:26.221 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T10:51:26.224 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id=33 2023-10-11T10:51:26.224 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 33 = 33 2023-10-11T10:51:26.224 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T10:51:26.224 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_snap_present cluster1 mirror test snap8 2023-10-11T10:51:26.224 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T10:51:26.225 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:51:26.225 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:51:26.225 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap8 2023-10-11T10:51:26.225 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T10:51:26.225 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-10-11T10:51:26.226 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 1 2023-10-11T10:51:27.224 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 info mirror/test@snap8 2023-10-11T10:51:27.276 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:rbd image 'test': 2023-10-11T10:51:27.276 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: size 512 MiB in 128 objects 2023-10-11T10:51:27.277 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: order 22 (4 MiB objects) 2023-10-11T10:51:27.277 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: snapshot_count: 16 2023-10-11T10:51:27.277 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: id: 10afefb6b972 2023-10-11T10:51:27.277 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: block_name_prefix: rbd_data.10afefb6b972 2023-10-11T10:51:27.277 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: format: 2 2023-10-11T10:51:27.278 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-10-11T10:51:27.278 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: op_features: 2023-10-11T10:51:27.278 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: flags: 2023-10-11T10:51:27.278 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: create_timestamp: Wed Oct 11 10:44:53 2023 2023-10-11T10:51:27.278 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: access_timestamp: Wed Oct 11 10:49:46 2023 2023-10-11T10:51:27.278 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: modify_timestamp: Wed Oct 11 10:44:53 2023 2023-10-11T10:51:27.279 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: protected: False 2023-10-11T10:51:27.279 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring state: enabled 2023-10-11T10:51:27.279 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring mode: snapshot 2023-10-11T10:51:27.279 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring global id: 3dd93db4-c9cc-462c-8b49-007eba9a90d2 2023-10-11T10:51:27.279 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring primary: false 2023-10-11T10:51:27.280 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T10:51:27.281 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n snap7-clean ']' 2023-10-11T10:51:27.281 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ compare_image_snaps mirror test snap7-clean 2023-10-11T10:51:27.281 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:51:27.281 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:51:27.281 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap7-clean 2023-10-11T10:51:27.282 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local ret=0 2023-10-11T10:51:27.282 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local rmt_export=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export 2023-10-11T10:51:27.282 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local loc_export=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:51:27.282 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rm -f /tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:51:27.282 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 -p mirror export test@snap7-clean /tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export 2023-10-11T10:51:27.870 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-10-11T10:51:27.879 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 -p mirror export test@snap7-clean /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:51:28.608 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-10-11T10:51:28.618 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ cmp /tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:51:28.832 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rm -f /tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:51:28.974 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T10:51:28.974 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ compare_image_snaps mirror test snap8 2023-10-11T10:51:28.974 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:51:28.974 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:51:28.975 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap8 2023-10-11T10:51:28.975 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local ret=0 2023-10-11T10:51:28.975 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local rmt_export=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export 2023-10-11T10:51:28.975 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local loc_export=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:51:28.975 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rm -f /tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:51:28.976 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 -p mirror export test@snap8 /tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export 2023-10-11T10:51:29.539 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-10-11T10:51:29.549 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 -p mirror export test@snap8 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:51:30.288 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-10-11T10:51:30.297 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ cmp /tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:51:30.511 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rm -f /tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:51:30.652 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T10:51:30.652 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ clean_snap_name=snap8-clean 2023-10-11T10:51:30.652 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ create_snap cluster2 mirror test snap8-clean 2023-10-11T10:51:30.653 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:51:30.653 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:51:30.653 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:51:30.653 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap8-clean 2023-10-11T10:51:30.653 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap8-clean --debug-rbd=20 --debug-journaler=20 2023-10-11T10:51:30.931 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 10` 2023-10-11T10:51:30.931 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ stress_write_image cluster2 mirror test 2023-10-11T10:51:30.931 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:51:30.932 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:51:30.932 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:51:30.932 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-10-11T10:51:30.932 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local duration=9 2023-10-11T10:51:30.933 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ set +e 2023-10-11T10:51:30.933 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2023-10-11T10:51:31.261 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:100 IOs, 60400640 bytes 2023-10-11T10:51:31.534 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:200 IOs, 160823296 bytes 2023-10-11T10:51:31.671 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:300 IOs, 170919936 bytes 2023-10-11T10:51:31.877 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:400 IOs, 233833472 bytes 2023-10-11T10:51:31.996 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:500 IOs, 253721600 bytes 2023-10-11T10:51:32.137 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:600 IOs, 287040512 bytes 2023-10-11T10:51:32.233 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:700 IOs, 287659008 bytes 2023-10-11T10:51:32.387 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:800 IOs, 332078080 bytes 2023-10-11T10:51:32.596 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:900 IOs, 396564480 bytes 2023-10-11T10:51:32.797 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1000 IOs, 475611136 bytes 2023-10-11T10:51:32.928 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1100 IOs, 485554176 bytes 2023-10-11T10:51:33.088 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1200 IOs, 530589696 bytes 2023-10-11T10:51:33.176 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1300 IOs, 531410944 bytes 2023-10-11T10:51:33.370 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1400 IOs, 597877760 bytes 2023-10-11T10:51:33.533 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1500 IOs, 642908160 bytes 2023-10-11T10:51:33.622 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1600 IOs, 653171712 bytes 2023-10-11T10:51:33.829 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1700 IOs, 717428736 bytes 2023-10-11T10:51:33.985 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1800 IOs, 755891200 bytes 2023-10-11T10:51:34.150 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1900 IOs, 793166848 bytes 2023-10-11T10:51:34.229 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2000 IOs, 793786368 bytes 2023-10-11T10:51:34.315 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2100 IOs, 798395392 bytes 2023-10-11T10:51:34.442 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2200 IOs, 835111936 bytes 2023-10-11T10:51:34.654 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2300 IOs, 890959872 bytes 2023-10-11T10:51:34.740 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2400 IOs, 893267968 bytes 2023-10-11T10:51:34.987 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2500 IOs, 983130112 bytes 2023-10-11T10:51:35.112 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2600 IOs, 1010478080 bytes 2023-10-11T10:51:35.234 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2700 IOs, 1032564736 bytes 2023-10-11T10:51:35.314 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2800 IOs, 1033185280 bytes 2023-10-11T10:51:35.452 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2900 IOs, 1064828928 bytes 2023-10-11T10:51:35.515 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.3502710041590157) > weight (0.25) 2023-10-11T10:51:35.516 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.5310408791370684) > weight (0.25) 2023-10-11T10:51:35.516 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.2 2023-10-11T10:51:35.517 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Sent signal 15 2023-10-11T10:51:35.517 INFO:tasks.rbd_mirror_thrash:waiting for 9 secs before reviving daemons 2023-10-11T10:51:35.517 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi111.stderr:2023-10-11T10:51:35.516+0000 7fbb420b6700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 (PID: 106603) UID: 1000 2023-10-11T10:51:35.599 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3000 IOs, 1093690368 bytes 2023-10-11T10:51:35.703 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3100 IOs, 1108879360 bytes 2023-10-11T10:51:35.806 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3200 IOs, 1121267712 bytes 2023-10-11T10:51:35.910 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3300 IOs, 1136828416 bytes 2023-10-11T10:51:36.089 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3400 IOs, 1192512512 bytes 2023-10-11T10:51:36.268 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3500 IOs, 1249519616 bytes 2023-10-11T10:51:36.452 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3600 IOs, 1300074496 bytes 2023-10-11T10:51:36.607 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3700 IOs, 1340445696 bytes 2023-10-11T10:51:36.831 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3800 IOs, 1437166592 bytes 2023-10-11T10:51:37.073 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3900 IOs, 1509787648 bytes 2023-10-11T10:51:37.276 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4000 IOs, 1566267392 bytes 2023-10-11T10:51:37.401 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4100 IOs, 1602572288 bytes 2023-10-11T10:51:37.602 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4200 IOs, 1651495936 bytes 2023-10-11T10:51:37.773 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4300 IOs, 1700361216 bytes 2023-10-11T10:51:37.953 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4400 IOs, 1777415168 bytes 2023-10-11T10:51:38.197 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4500 IOs, 1836253184 bytes 2023-10-11T10:51:38.373 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4600 IOs, 1885442048 bytes 2023-10-11T10:51:38.581 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4700 IOs, 1946270720 bytes 2023-10-11T10:51:38.659 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4800 IOs, 1948110848 bytes 2023-10-11T10:51:38.914 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4900 IOs, 2033892352 bytes 2023-10-11T10:51:39.074 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5000 IOs, 2090957824 bytes 2023-10-11T10:51:39.281 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5100 IOs, 2147162112 bytes 2023-10-11T10:51:39.451 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5200 IOs, 2191492096 bytes 2023-10-11T10:51:39.574 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5300 IOs, 2221982720 bytes 2023-10-11T10:51:39.679 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5400 IOs, 2224983040 bytes 2023-10-11T10:51:39.825 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5500 IOs, 2256730112 bytes 2023-10-11T10:51:39.930 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5600 IOs, 2291950592 bytes 2023-10-11T10:51:39.946 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ error_code=124 2023-10-11T10:51:39.947 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ set -e 2023-10-11T10:51:39.947 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' 124 -eq 124 ']' 2023-10-11T10:51:39.947 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T10:51:39.947 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2023-10-11T10:51:39.947 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T10:51:39.947 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:51:39.948 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:51:39.948 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local state_pattern=up+replaying 2023-10-11T10:51:39.948 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local description_pattern= 2023-10-11T10:51:39.948 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local service_pattern= 2023-10-11T10:51:39.948 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-10-11T10:51:39.948 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 1 2023-10-11T10:51:40.948 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2023-10-11T10:51:40.948 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T10:51:40.949 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:51:40.949 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:51:40.949 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local state_pattern=up+replaying 2023-10-11T10:51:40.949 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local description_pattern= 2023-10-11T10:51:40.949 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local service_pattern= 2023-10-11T10:51:40.950 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ mkfname cluster1-mirror-test.mirror_status 2023-10-11T10:51:40.950 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ echo cluster1-mirror-test.mirror_status 2023-10-11T10:51:40.950 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -e 's|[/ ]|_|g' 2023-10-11T10:51:40.951 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local status_log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.mirror_status 2023-10-11T10:51:40.951 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ CEPH_ARGS= 2023-10-11T10:51:40.951 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2023-10-11T10:51:40.951 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ tee /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.mirror_status 2023-10-11T10:51:41.003 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:test: 2023-10-11T10:51:41.004 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: global_id: 3dd93db4-c9cc-462c-8b49-007eba9a90d2 2023-10-11T10:51:41.004 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: state: up+replaying 2023-10-11T10:51:41.004 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":533857280.0,"last_snapshot_bytes":534917120,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1697021433,"remote_snapshot_timestamp":1697021433,"replay_state":"idle"} 2023-10-11T10:51:41.004 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: service: mirror.1 on smithi111 2023-10-11T10:51:41.004 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: last_update: 2023-10-11 10:51:21 2023-10-11T10:51:41.004 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: peer_sites: 2023-10-11T10:51:41.005 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: name: cluster2 2023-10-11T10:51:41.005 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: state: up+stopped 2023-10-11T10:51:41.005 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: description: local image is primary 2023-10-11T10:51:41.005 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: last_update: 2023-10-11 10:51:21 2023-10-11T10:51:41.032 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.mirror_status 2023-10-11T10:51:41.033 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: state: up+replaying 2023-10-11T10:51:41.034 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ grep '^ description: .*' /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.mirror_status 2023-10-11T10:51:41.034 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":533857280.0,"last_snapshot_bytes":534917120,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1697021433,"remote_snapshot_timestamp":1697021433,"replay_state":"idle"} 2023-10-11T10:51:41.035 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n '' ']' 2023-10-11T10:51:41.035 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ echo up+replaying 2023-10-11T10:51:41.036 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ grep '^up+' 2023-10-11T10:51:41.036 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:up+replaying 2023-10-11T10:51:41.037 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ grep 'service: *mirror..* on ' /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.mirror_status 2023-10-11T10:51:41.037 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: service: mirror.1 on smithi111 2023-10-11T10:51:41.037 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.mirror_status 2023-10-11T10:51:41.039 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local 'last_update=2023-10-11 10:51:21' 2023-10-11T10:51:41.039 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2023-10-11 10:51:21' 2023-10-11T10:51:41.039 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T10:51:41.040 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:51:41.040 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:51:41.040 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local state_pattern=up+replaying 2023-10-11T10:51:41.040 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local 'prev_last_update=2023-10-11 10:51:21' 2023-10-11T10:51:41.040 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ mkfname cluster1-mirror.mirror_status 2023-10-11T10:51:41.041 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ echo cluster1-mirror.mirror_status 2023-10-11T10:51:41.041 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -e 's|[/ ]|_|g' 2023-10-11T10:51:41.041 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local status_log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror.mirror_status 2023-10-11T10:51:41.042 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2023-10-11T10:51:41.080 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd: failed to query services: (13) Permission denied 2023-10-11T10:51:41.116 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local last_update state 2023-10-11T10:51:41.117 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2023-10-11T10:51:41.119 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ last_update='2023-10-11 10:51:21' 2023-10-11T10:51:41.119 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2023-10-11T10:51:41.121 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ state=up+replaying 2023-10-11T10:51:41.121 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ echo up+replaying 2023-10-11T10:51:41.121 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ grep up+replaying 2023-10-11T10:51:41.122 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:up+replaying 2023-10-11T10:51:41.123 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T10:51:41.123 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T10:51:41.123 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ snap_name=snap9 2023-10-11T10:51:41.123 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ create_snap cluster2 mirror test snap9 2023-10-11T10:51:41.124 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:51:41.124 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:51:41.124 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:51:41.124 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap9 2023-10-11T10:51:41.124 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap9 --debug-rbd=20 --debug-journaler=20 2023-10-11T10:51:44.517 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.2 2023-10-11T10:51:44.518 INFO:teuthology.orchestra.run:waiting for 600 2023-10-11T10:51:44.518 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.2 2023-10-11T10:51:44.518 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Restarting daemon 2023-10-11T10:51:44.519 DEBUG:teuthology.orchestra.run.smithi111:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 2023-10-11T10:51:44.522 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Started 2023-10-11T10:51:44.522 INFO:tasks.rbd_mirror_thrash:waiting for 95 secs before thrashing 2023-10-11T10:51:56.517 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.2556116495004962) > weight (0.25) 2023-10-11T10:51:56.517 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.1 2023-10-11T10:51:56.518 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Sent signal 15 2023-10-11T10:51:56.518 INFO:tasks.rbd_mirror_thrash:waiting for 9 secs before reviving daemons 2023-10-11T10:51:56.519 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi111.stderr:2023-10-11T10:51:56.518+0000 7f8157698700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 110429) UID: 1000 2023-10-11T10:52:05.520 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.1 2023-10-11T10:52:05.520 INFO:teuthology.orchestra.run:waiting for 600 2023-10-11T10:52:05.521 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.1 2023-10-11T10:52:05.521 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2023-10-11T10:52:05.521 DEBUG:teuthology.orchestra.run.smithi111:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2023-10-11T10:52:05.524 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2023-10-11T10:52:05.524 INFO:tasks.rbd_mirror_thrash:waiting for 98 secs before thrashing 2023-10-11T10:52:07.938 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_replay_started cluster1 mirror test 2023-10-11T10:52:07.939 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T10:52:07.939 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:52:07.939 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:52:07.939 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2023-10-11T10:52:07.940 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T10:52:07.940 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:52:07.940 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:52:07.940 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local state=started 2023-10-11T10:52:07.941 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T10:52:07.941 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-10-11T10:52:07.941 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 1 2023-10-11T10:52:08.938 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test_image_replay_state cluster1 mirror test started 2023-10-11T10:52:08.939 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T10:52:08.939 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:52:08.939 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:52:08.939 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local test_state=started 2023-10-11T10:52:08.940 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local status_result 2023-10-11T10:52:08.940 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_state=stopped 2023-10-11T10:52:08.944 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2023-10-11T10:52:08.944 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_instance=cluster1 2023-10-11T10:52:08.944 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ grep -i state 2023-10-11T10:52:08.944 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T10:52:08.945 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T10:52:08.945 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance=cluster1 2023-10-11T10:52:08.945 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local loop_instance 2023-10-11T10:52:08.945 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-10-11T10:52:08.945 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sleep 0 2023-10-11T10:52:08.946 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-10-11T10:52:08.946 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ seq 0 3 2023-10-11T10:52:08.946 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-10-11T10:52:08.946 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-10-11T10:52:08.946 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1:0 2023-10-11T10:52:08.946 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T10:52:08.947 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance 2023-10-11T10:52:08.947 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-10-11T10:52:08.947 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local val=cluster1:0 2023-10-11T10:52:08.947 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_var_name=cluster 2023-10-11T10:52:08.947 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance_var_name=instance 2023-10-11T10:52:08.948 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ cluster=cluster1 2023-10-11T10:52:08.948 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ instance=0 2023-10-11T10:52:08.948 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' 0 = cluster1:0 ']' 2023-10-11T10:52:08.948 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval cluster=cluster1 2023-10-11T10:52:08.948 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T10:52:08.949 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval instance=0 2023-10-11T10:52:08.949 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=0 2023-10-11T10:52:08.949 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-10-11T10:52:08.949 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local local_cluster=cluster1:0 2023-10-11T10:52:08.949 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster=cluster1 2023-10-11T10:52:08.950 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance 2023-10-11T10:52:08.950 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-10-11T10:52:08.950 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local val=cluster1:0 2023-10-11T10:52:08.950 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster_var_name=local_cluster 2023-10-11T10:52:08.951 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance_var_name=instance 2023-10-11T10:52:08.951 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T10:52:08.951 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=0 2023-10-11T10:52:08.951 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ '[' 0 = cluster1:0 ']' 2023-10-11T10:52:08.951 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval local_cluster=cluster1 2023-10-11T10:52:08.951 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ local_cluster=cluster1 2023-10-11T10:52:08.952 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval instance=0 2023-10-11T10:52:08.952 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ instance=0 2023-10-11T10:52:08.952 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-10-11T10:52:08.965 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T10:52:08.965 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T10:52:08.966 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T10:52:08.966 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-10-11T10:52:09.065 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-10-11T10:52:09.065 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd mirror status : get status for rbd mirror 2023-10-11T10:52:09.066 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:admin_socket: invalid command 2023-10-11T10:52:09.068 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-10-11T10:52:09.069 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-10-11T10:52:09.069 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1:1 2023-10-11T10:52:09.069 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T10:52:09.069 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance 2023-10-11T10:52:09.069 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-10-11T10:52:09.070 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local val=cluster1:1 2023-10-11T10:52:09.070 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_var_name=cluster 2023-10-11T10:52:09.070 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance_var_name=instance 2023-10-11T10:52:09.070 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ cluster=cluster1 2023-10-11T10:52:09.070 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ instance=1 2023-10-11T10:52:09.071 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' 1 = cluster1:1 ']' 2023-10-11T10:52:09.071 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval cluster=cluster1 2023-10-11T10:52:09.071 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T10:52:09.071 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval instance=1 2023-10-11T10:52:09.071 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=1 2023-10-11T10:52:09.072 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-10-11T10:52:09.072 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local local_cluster=cluster1:1 2023-10-11T10:52:09.072 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster=cluster1 2023-10-11T10:52:09.072 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance 2023-10-11T10:52:09.072 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-10-11T10:52:09.072 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local val=cluster1:1 2023-10-11T10:52:09.073 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster_var_name=local_cluster 2023-10-11T10:52:09.073 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance_var_name=instance 2023-10-11T10:52:09.073 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T10:52:09.073 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=1 2023-10-11T10:52:09.073 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ '[' 1 = cluster1:1 ']' 2023-10-11T10:52:09.073 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval local_cluster=cluster1 2023-10-11T10:52:09.074 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ local_cluster=cluster1 2023-10-11T10:52:09.074 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval instance=1 2023-10-11T10:52:09.074 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ instance=1 2023-10-11T10:52:09.074 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-10-11T10:52:09.087 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T10:52:09.087 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T10:52:09.088 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T10:52:09.088 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-10-11T10:52:09.192 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ return 0 2023-10-11T10:52:09.193 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ status_result=' "state": "Replaying"' 2023-10-11T10:52:09.193 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ echo ' "state": "Replaying"' 2023-10-11T10:52:09.193 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ grep -i Replaying 2023-10-11T10:52:09.194 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: "state": "Replaying" 2023-10-11T10:52:09.196 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ current_state=started 2023-10-11T10:52:09.197 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test started = started 2023-10-11T10:52:09.197 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T10:52:09.197 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2023-10-11T10:52:09.197 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local local_cluster=cluster1 2023-10-11T10:52:09.197 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:52:09.197 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:52:09.198 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:52:09.198 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' snapshot = journal ']' 2023-10-11T10:52:09.198 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' snapshot = snapshot ']' 2023-10-11T10:52:09.198 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2023-10-11T10:52:09.198 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local local_cluster=cluster1 2023-10-11T10:52:09.199 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:52:09.199 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:52:09.199 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:52:09.199 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ mkfname cluster2-mirror-test.status 2023-10-11T10:52:09.199 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ echo cluster2-mirror-test.status 2023-10-11T10:52:09.200 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -e 's|[/ ]|_|g' 2023-10-11T10:52:09.200 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local status_log=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.status 2023-10-11T10:52:09.200 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ mkfname cluster1-mirror-test.status 2023-10-11T10:52:09.200 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ echo cluster1-mirror-test.status 2023-10-11T10:52:09.200 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -e 's|[/ ]|_|g' 2023-10-11T10:52:09.201 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local local_status_log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.status 2023-10-11T10:52:09.202 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ mirror_image_snapshot cluster2 mirror test 2023-10-11T10:52:09.202 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:52:09.202 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:52:09.202 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:52:09.202 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2023-10-11T10:52:10.936 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:Snapshot ID: 37 2023-10-11T10:52:10.962 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.status 2023-10-11T10:52:10.962 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:52:10.962 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:52:10.963 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:52:10.963 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.status 2023-10-11T10:52:10.963 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2023-10-11T10:52:10.963 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T10:52:11.035 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-10-11T10:52:11.038 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snapshot_id=37 2023-10-11T10:52:11.038 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T10:52:11.038 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T10:52:11.038 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.2 2023-10-11T10:52:11.239 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.status 2023-10-11T10:52:11.239 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T10:52:11.240 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:52:11.240 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:52:11.240 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.status 2023-10-11T10:52:11.240 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-10-11T10:52:11.240 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T10:52:11.286 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T10:52:11.287 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T10:52:11.288 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:-:1.1: Document is empty 2023-10-11T10:52:11.289 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: 2023-10-11T10:52:11.289 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:^ 2023-10-11T10:52:11.289 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id= 2023-10-11T10:52:11.289 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 37 = '' 2023-10-11T10:52:11.289 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T10:52:11.290 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.4 2023-10-11T10:52:11.690 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.status 2023-10-11T10:52:11.690 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T10:52:11.690 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:52:11.691 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:52:11.691 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.status 2023-10-11T10:52:11.691 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-10-11T10:52:11.691 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T10:52:11.738 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T10:52:11.739 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T10:52:11.740 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:-:1.1: Document is empty 2023-10-11T10:52:11.740 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: 2023-10-11T10:52:11.740 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:^ 2023-10-11T10:52:11.741 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id= 2023-10-11T10:52:11.741 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 37 = '' 2023-10-11T10:52:11.741 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T10:52:11.742 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.8 2023-10-11T10:52:12.542 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.status 2023-10-11T10:52:12.542 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T10:52:12.542 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:52:12.543 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:52:12.543 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.status 2023-10-11T10:52:12.543 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-10-11T10:52:12.543 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T10:52:12.596 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T10:52:12.596 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T10:52:12.597 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:-:1.1: Document is empty 2023-10-11T10:52:12.598 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: 2023-10-11T10:52:12.598 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:^ 2023-10-11T10:52:12.598 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id= 2023-10-11T10:52:12.598 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 37 = '' 2023-10-11T10:52:12.598 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T10:52:12.599 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 1.6 2023-10-11T10:52:14.199 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.status 2023-10-11T10:52:14.200 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T10:52:14.200 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:52:14.200 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:52:14.200 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.status 2023-10-11T10:52:14.200 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-10-11T10:52:14.201 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T10:52:14.250 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T10:52:14.253 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id=37 2023-10-11T10:52:14.253 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 37 = 37 2023-10-11T10:52:14.253 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T10:52:14.253 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_snap_present cluster1 mirror test snap9 2023-10-11T10:52:14.253 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T10:52:14.254 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:52:14.254 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:52:14.254 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap9 2023-10-11T10:52:14.254 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T10:52:14.254 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-10-11T10:52:14.255 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 1 2023-10-11T10:52:15.254 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 info mirror/test@snap9 2023-10-11T10:52:15.302 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:rbd image 'test': 2023-10-11T10:52:15.303 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: size 512 MiB in 128 objects 2023-10-11T10:52:15.303 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: order 22 (4 MiB objects) 2023-10-11T10:52:15.303 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: snapshot_count: 18 2023-10-11T10:52:15.303 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: id: 10afefb6b972 2023-10-11T10:52:15.303 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: block_name_prefix: rbd_data.10afefb6b972 2023-10-11T10:52:15.304 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: format: 2 2023-10-11T10:52:15.304 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-10-11T10:52:15.304 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: op_features: 2023-10-11T10:52:15.304 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: flags: 2023-10-11T10:52:15.304 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: create_timestamp: Wed Oct 11 10:44:53 2023 2023-10-11T10:52:15.305 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: access_timestamp: Wed Oct 11 10:51:27 2023 2023-10-11T10:52:15.305 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: modify_timestamp: Wed Oct 11 10:44:53 2023 2023-10-11T10:52:15.305 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: protected: False 2023-10-11T10:52:15.305 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring state: enabled 2023-10-11T10:52:15.305 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring mode: snapshot 2023-10-11T10:52:15.306 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring global id: 3dd93db4-c9cc-462c-8b49-007eba9a90d2 2023-10-11T10:52:15.306 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring primary: false 2023-10-11T10:52:15.307 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T10:52:15.307 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n snap8-clean ']' 2023-10-11T10:52:15.307 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ compare_image_snaps mirror test snap8-clean 2023-10-11T10:52:15.308 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:52:15.308 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:52:15.308 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap8-clean 2023-10-11T10:52:15.308 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local ret=0 2023-10-11T10:52:15.308 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local rmt_export=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export 2023-10-11T10:52:15.309 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local loc_export=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:52:15.309 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rm -f /tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:52:15.309 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 -p mirror export test@snap8-clean /tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export 2023-10-11T10:52:15.873 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-10-11T10:52:15.879 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 -p mirror export test@snap8-clean /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:52:16.617 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-10-11T10:52:16.626 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ cmp /tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:52:16.844 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rm -f /tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:52:16.986 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T10:52:16.986 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ compare_image_snaps mirror test snap9 2023-10-11T10:52:16.986 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:52:16.986 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:52:16.987 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap9 2023-10-11T10:52:16.987 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local ret=0 2023-10-11T10:52:16.987 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local rmt_export=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export 2023-10-11T10:52:16.987 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local loc_export=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:52:16.987 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rm -f /tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:52:16.988 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 -p mirror export test@snap9 /tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export 2023-10-11T10:52:17.770 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-10-11T10:52:17.779 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 -p mirror export test@snap9 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:52:18.496 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-10-11T10:52:18.504 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ cmp /tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:52:18.718 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rm -f /tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:52:18.859 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T10:52:18.859 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ clean_snap_name=snap9-clean 2023-10-11T10:52:18.859 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ create_snap cluster2 mirror test snap9-clean 2023-10-11T10:52:18.859 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:52:18.860 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:52:18.860 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:52:18.860 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap9-clean 2023-10-11T10:52:18.860 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap9-clean --debug-rbd=20 --debug-journaler=20 2023-10-11T10:52:19.909 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 10` 2023-10-11T10:52:19.909 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ stress_write_image cluster2 mirror test 2023-10-11T10:52:19.909 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:52:19.910 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:52:19.910 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:52:19.910 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-10-11T10:52:19.911 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local duration=6 2023-10-11T10:52:19.911 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ set +e 2023-10-11T10:52:19.911 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2023-10-11T10:52:20.317 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:100 IOs, 89119744 bytes 2023-10-11T10:52:20.501 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:200 IOs, 116002816 bytes 2023-10-11T10:52:20.706 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:300 IOs, 172152832 bytes 2023-10-11T10:52:20.823 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:400 IOs, 190774272 bytes 2023-10-11T10:52:21.068 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:500 IOs, 279602176 bytes 2023-10-11T10:52:21.223 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:600 IOs, 323259392 bytes 2023-10-11T10:52:21.307 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:700 IOs, 324111360 bytes 2023-10-11T10:52:21.497 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:800 IOs, 382150656 bytes 2023-10-11T10:52:21.645 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:900 IOs, 435052544 bytes 2023-10-11T10:52:21.881 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1000 IOs, 504587264 bytes 2023-10-11T10:52:21.987 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1100 IOs, 514214912 bytes 2023-10-11T10:52:22.140 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1200 IOs, 565169152 bytes 2023-10-11T10:52:22.257 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1300 IOs, 583908352 bytes 2023-10-11T10:52:22.363 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1400 IOs, 603214848 bytes 2023-10-11T10:52:22.494 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1500 IOs, 646001664 bytes 2023-10-11T10:52:22.628 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1600 IOs, 646643712 bytes 2023-10-11T10:52:22.752 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1700 IOs, 671554560 bytes 2023-10-11T10:52:22.954 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1800 IOs, 734637056 bytes 2023-10-11T10:52:23.121 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1900 IOs, 779564032 bytes 2023-10-11T10:52:23.211 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2000 IOs, 791830528 bytes 2023-10-11T10:52:23.431 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2100 IOs, 856309760 bytes 2023-10-11T10:52:23.654 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2200 IOs, 930683904 bytes 2023-10-11T10:52:23.825 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2300 IOs, 980698112 bytes 2023-10-11T10:52:23.937 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2400 IOs, 998161408 bytes 2023-10-11T10:52:24.069 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2500 IOs, 1027404800 bytes 2023-10-11T10:52:24.176 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2600 IOs, 1039652864 bytes 2023-10-11T10:52:24.268 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2700 IOs, 1044701184 bytes 2023-10-11T10:52:24.402 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2800 IOs, 1068690432 bytes 2023-10-11T10:52:24.615 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2900 IOs, 1154988032 bytes 2023-10-11T10:52:24.849 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3000 IOs, 1220145152 bytes 2023-10-11T10:52:24.996 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3100 IOs, 1245062144 bytes 2023-10-11T10:52:25.172 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3200 IOs, 1297518592 bytes 2023-10-11T10:52:25.306 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3300 IOs, 1332488192 bytes 2023-10-11T10:52:25.408 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3400 IOs, 1339598848 bytes 2023-10-11T10:52:25.661 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3500 IOs, 1426830336 bytes 2023-10-11T10:52:25.859 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3600 IOs, 1487508480 bytes 2023-10-11T10:52:25.916 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ error_code=124 2023-10-11T10:52:25.917 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ set -e 2023-10-11T10:52:25.917 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' 124 -eq 124 ']' 2023-10-11T10:52:25.917 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T10:52:25.917 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2023-10-11T10:52:25.917 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T10:52:25.918 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:52:25.918 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:52:25.918 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local state_pattern=up+replaying 2023-10-11T10:52:25.918 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local description_pattern= 2023-10-11T10:52:25.918 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local service_pattern= 2023-10-11T10:52:25.919 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-10-11T10:52:25.919 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 1 2023-10-11T10:52:26.918 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2023-10-11T10:52:26.918 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T10:52:26.918 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:52:26.919 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:52:26.919 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local state_pattern=up+replaying 2023-10-11T10:52:26.919 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local description_pattern= 2023-10-11T10:52:26.919 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local service_pattern= 2023-10-11T10:52:26.919 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ mkfname cluster1-mirror-test.mirror_status 2023-10-11T10:52:26.920 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ echo cluster1-mirror-test.mirror_status 2023-10-11T10:52:26.920 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -e 's|[/ ]|_|g' 2023-10-11T10:52:26.920 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local status_log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.mirror_status 2023-10-11T10:52:26.921 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ CEPH_ARGS= 2023-10-11T10:52:26.921 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2023-10-11T10:52:26.921 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ tee /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.mirror_status 2023-10-11T10:52:27.174 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:test: 2023-10-11T10:52:27.175 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: global_id: 3dd93db4-c9cc-462c-8b49-007eba9a90d2 2023-10-11T10:52:27.175 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: state: up+replaying 2023-10-11T10:52:27.175 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: description: replaying, {"bytes_per_second":17402026.67,"bytes_per_snapshot":496872448.0,"last_snapshot_bytes":522060800,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1697021529,"remote_snapshot_timestamp":1697021529,"replay_state":"idle"} 2023-10-11T10:52:27.175 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: service: mirror.1 on smithi111 2023-10-11T10:52:27.176 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: last_update: 2023-10-11 10:52:21 2023-10-11T10:52:27.176 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: peer_sites: 2023-10-11T10:52:27.176 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: name: cluster2 2023-10-11T10:52:27.176 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: state: up+stopped 2023-10-11T10:52:27.176 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: description: local image is primary 2023-10-11T10:52:27.177 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: last_update: 2023-10-11 10:52:21 2023-10-11T10:52:27.180 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.mirror_status 2023-10-11T10:52:27.181 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: state: up+replaying 2023-10-11T10:52:27.182 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ grep '^ description: .*' /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.mirror_status 2023-10-11T10:52:27.182 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: description: replaying, {"bytes_per_second":17402026.67,"bytes_per_snapshot":496872448.0,"last_snapshot_bytes":522060800,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1697021529,"remote_snapshot_timestamp":1697021529,"replay_state":"idle"} 2023-10-11T10:52:27.183 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n '' ']' 2023-10-11T10:52:27.183 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ echo up+replaying 2023-10-11T10:52:27.184 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ grep '^up+' 2023-10-11T10:52:27.184 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:up+replaying 2023-10-11T10:52:27.184 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ grep 'service: *mirror..* on ' /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.mirror_status 2023-10-11T10:52:27.185 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: service: mirror.1 on smithi111 2023-10-11T10:52:27.185 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.mirror_status 2023-10-11T10:52:27.187 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local 'last_update=2023-10-11 10:52:21' 2023-10-11T10:52:27.187 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2023-10-11 10:52:21' 2023-10-11T10:52:27.188 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T10:52:27.188 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:52:27.188 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:52:27.188 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local state_pattern=up+replaying 2023-10-11T10:52:27.188 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local 'prev_last_update=2023-10-11 10:52:21' 2023-10-11T10:52:27.189 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ mkfname cluster1-mirror.mirror_status 2023-10-11T10:52:27.189 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ echo cluster1-mirror.mirror_status 2023-10-11T10:52:27.189 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -e 's|[/ ]|_|g' 2023-10-11T10:52:27.190 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local status_log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror.mirror_status 2023-10-11T10:52:27.190 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2023-10-11T10:52:27.228 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd: failed to query services: (13) Permission denied 2023-10-11T10:52:27.252 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local last_update state 2023-10-11T10:52:27.252 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2023-10-11T10:52:27.254 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ last_update='2023-10-11 10:52:21' 2023-10-11T10:52:27.255 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2023-10-11T10:52:27.257 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ state=up+replaying 2023-10-11T10:52:27.257 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ echo up+replaying 2023-10-11T10:52:27.257 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ grep up+replaying 2023-10-11T10:52:27.257 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:up+replaying 2023-10-11T10:52:27.258 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T10:52:27.259 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T10:52:27.259 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ snap_name=snap10 2023-10-11T10:52:27.259 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ create_snap cluster2 mirror test snap10 2023-10-11T10:52:27.259 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:52:27.259 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:52:27.260 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:52:27.260 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap10 2023-10-11T10:52:27.260 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap10 --debug-rbd=20 --debug-journaler=20 2023-10-11T10:52:56.948 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_replay_started cluster1 mirror test 2023-10-11T10:52:56.948 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T10:52:56.948 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:52:56.949 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:52:56.949 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2023-10-11T10:52:56.949 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T10:52:56.949 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:52:56.949 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:52:56.950 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local state=started 2023-10-11T10:52:56.950 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T10:52:56.950 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-10-11T10:52:56.950 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 1 2023-10-11T10:52:57.949 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test_image_replay_state cluster1 mirror test started 2023-10-11T10:52:57.949 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T10:52:57.949 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:52:57.949 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:52:57.950 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local test_state=started 2023-10-11T10:52:57.950 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local status_result 2023-10-11T10:52:57.950 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_state=stopped 2023-10-11T10:52:57.954 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2023-10-11T10:52:57.954 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_instance=cluster1 2023-10-11T10:52:57.954 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T10:52:57.954 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T10:52:57.955 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ grep -i state 2023-10-11T10:52:57.955 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance=cluster1 2023-10-11T10:52:57.955 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local loop_instance 2023-10-11T10:52:57.955 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-10-11T10:52:57.956 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sleep 0 2023-10-11T10:52:57.956 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-10-11T10:52:57.956 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ seq 0 3 2023-10-11T10:52:57.956 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-10-11T10:52:57.956 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-10-11T10:52:57.957 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1:0 2023-10-11T10:52:57.957 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T10:52:57.957 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance 2023-10-11T10:52:57.957 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-10-11T10:52:57.958 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local val=cluster1:0 2023-10-11T10:52:57.958 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_var_name=cluster 2023-10-11T10:52:57.958 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance_var_name=instance 2023-10-11T10:52:57.958 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ cluster=cluster1 2023-10-11T10:52:57.958 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ instance=0 2023-10-11T10:52:57.959 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' 0 = cluster1:0 ']' 2023-10-11T10:52:57.959 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval cluster=cluster1 2023-10-11T10:52:57.959 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T10:52:57.959 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval instance=0 2023-10-11T10:52:57.960 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=0 2023-10-11T10:52:57.960 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-10-11T10:52:57.960 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local local_cluster=cluster1:0 2023-10-11T10:52:57.960 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster=cluster1 2023-10-11T10:52:57.960 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance 2023-10-11T10:52:57.961 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-10-11T10:52:57.961 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local val=cluster1:0 2023-10-11T10:52:57.961 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster_var_name=local_cluster 2023-10-11T10:52:57.961 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance_var_name=instance 2023-10-11T10:52:57.961 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T10:52:57.962 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=0 2023-10-11T10:52:57.962 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ '[' 0 = cluster1:0 ']' 2023-10-11T10:52:57.962 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval local_cluster=cluster1 2023-10-11T10:52:57.962 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ local_cluster=cluster1 2023-10-11T10:52:57.962 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval instance=0 2023-10-11T10:52:57.963 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ instance=0 2023-10-11T10:52:57.963 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-10-11T10:52:57.976 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T10:52:57.976 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T10:52:57.976 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T10:52:57.976 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-10-11T10:52:58.076 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-10-11T10:52:58.076 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd mirror status : get status for rbd mirror 2023-10-11T10:52:58.077 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:admin_socket: invalid command 2023-10-11T10:52:58.080 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-10-11T10:52:58.080 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-10-11T10:52:58.080 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1:1 2023-10-11T10:52:58.080 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T10:52:58.081 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance 2023-10-11T10:52:58.081 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-10-11T10:52:58.081 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local val=cluster1:1 2023-10-11T10:52:58.081 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_var_name=cluster 2023-10-11T10:52:58.081 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance_var_name=instance 2023-10-11T10:52:58.082 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ cluster=cluster1 2023-10-11T10:52:58.082 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ instance=1 2023-10-11T10:52:58.082 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' 1 = cluster1:1 ']' 2023-10-11T10:52:58.082 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval cluster=cluster1 2023-10-11T10:52:58.083 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T10:52:58.083 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval instance=1 2023-10-11T10:52:58.083 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=1 2023-10-11T10:52:58.083 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-10-11T10:52:58.084 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local local_cluster=cluster1:1 2023-10-11T10:52:58.084 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster=cluster1 2023-10-11T10:52:58.084 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance 2023-10-11T10:52:58.084 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-10-11T10:52:58.084 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local val=cluster1:1 2023-10-11T10:52:58.085 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster_var_name=local_cluster 2023-10-11T10:52:58.085 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance_var_name=instance 2023-10-11T10:52:58.085 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T10:52:58.085 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=1 2023-10-11T10:52:58.085 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ '[' 1 = cluster1:1 ']' 2023-10-11T10:52:58.086 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval local_cluster=cluster1 2023-10-11T10:52:58.086 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ local_cluster=cluster1 2023-10-11T10:52:58.086 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval instance=1 2023-10-11T10:52:58.086 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ instance=1 2023-10-11T10:52:58.087 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-10-11T10:52:58.098 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T10:52:58.099 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T10:52:58.099 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T10:52:58.099 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-10-11T10:52:58.203 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ return 0 2023-10-11T10:52:58.204 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ status_result=' "state": "Replaying"' 2023-10-11T10:52:58.204 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ echo ' "state": "Replaying"' 2023-10-11T10:52:58.205 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ grep -i Replaying 2023-10-11T10:52:58.205 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: "state": "Replaying" 2023-10-11T10:52:58.207 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ current_state=started 2023-10-11T10:52:58.208 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test started = started 2023-10-11T10:52:58.208 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T10:52:58.208 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2023-10-11T10:52:58.208 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local local_cluster=cluster1 2023-10-11T10:52:58.209 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:52:58.209 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:52:58.209 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:52:58.209 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' snapshot = journal ']' 2023-10-11T10:52:58.210 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' snapshot = snapshot ']' 2023-10-11T10:52:58.210 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2023-10-11T10:52:58.210 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local local_cluster=cluster1 2023-10-11T10:52:58.210 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:52:58.211 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:52:58.211 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:52:58.211 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ mkfname cluster2-mirror-test.status 2023-10-11T10:52:58.211 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ echo cluster2-mirror-test.status 2023-10-11T10:52:58.211 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -e 's|[/ ]|_|g' 2023-10-11T10:52:58.212 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local status_log=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.status 2023-10-11T10:52:58.212 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ mkfname cluster1-mirror-test.status 2023-10-11T10:52:58.212 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ echo cluster1-mirror-test.status 2023-10-11T10:52:58.212 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -e 's|[/ ]|_|g' 2023-10-11T10:52:58.213 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local local_status_log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.status 2023-10-11T10:52:58.214 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ mirror_image_snapshot cluster2 mirror test 2023-10-11T10:52:58.214 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:52:58.214 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:52:58.214 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:52:58.215 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2023-10-11T10:52:59.933 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:Snapshot ID: 41 2023-10-11T10:52:59.959 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.status 2023-10-11T10:52:59.959 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:52:59.959 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:52:59.959 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:52:59.960 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.status 2023-10-11T10:52:59.969 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2023-10-11T10:52:59.970 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T10:53:00.025 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-10-11T10:53:00.027 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snapshot_id=41 2023-10-11T10:53:00.027 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T10:53:00.028 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T10:53:00.028 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.2 2023-10-11T10:53:00.228 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.status 2023-10-11T10:53:00.229 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T10:53:00.229 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:53:00.229 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:53:00.229 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.status 2023-10-11T10:53:00.229 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-10-11T10:53:00.230 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T10:53:00.282 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T10:53:00.282 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T10:53:00.283 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:-:1.1: Document is empty 2023-10-11T10:53:00.283 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: 2023-10-11T10:53:00.283 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:^ 2023-10-11T10:53:00.284 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id= 2023-10-11T10:53:00.284 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 41 = '' 2023-10-11T10:53:00.285 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T10:53:00.285 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.4 2023-10-11T10:53:00.685 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.status 2023-10-11T10:53:00.685 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T10:53:00.685 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:53:00.686 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:53:00.686 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.status 2023-10-11T10:53:00.686 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-10-11T10:53:00.686 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T10:53:00.739 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T10:53:00.740 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T10:53:00.741 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:-:1.1: Document is empty 2023-10-11T10:53:00.741 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: 2023-10-11T10:53:00.742 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:^ 2023-10-11T10:53:00.742 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id= 2023-10-11T10:53:00.742 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 41 = '' 2023-10-11T10:53:00.743 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T10:53:00.743 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.8 2023-10-11T10:53:01.543 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.status 2023-10-11T10:53:01.543 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T10:53:01.543 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:53:01.544 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:53:01.544 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.status 2023-10-11T10:53:01.544 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-10-11T10:53:01.544 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T10:53:01.597 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T10:53:01.598 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T10:53:01.599 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:-:1.1: Document is empty 2023-10-11T10:53:01.600 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: 2023-10-11T10:53:01.600 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:^ 2023-10-11T10:53:01.600 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id= 2023-10-11T10:53:01.600 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 41 = '' 2023-10-11T10:53:01.601 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T10:53:01.601 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 1.6 2023-10-11T10:53:03.201 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.status 2023-10-11T10:53:03.202 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T10:53:03.202 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:53:03.202 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:53:03.202 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.status 2023-10-11T10:53:03.202 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-10-11T10:53:03.203 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T10:53:03.251 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T10:53:03.254 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id=41 2023-10-11T10:53:03.254 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 41 = 41 2023-10-11T10:53:03.254 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T10:53:03.254 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_snap_present cluster1 mirror test snap10 2023-10-11T10:53:03.254 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T10:53:03.255 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:53:03.255 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:53:03.255 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap10 2023-10-11T10:53:03.255 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T10:53:03.255 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-10-11T10:53:03.256 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 1 2023-10-11T10:53:04.254 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 info mirror/test@snap10 2023-10-11T10:53:04.304 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:rbd image 'test': 2023-10-11T10:53:04.304 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: size 512 MiB in 128 objects 2023-10-11T10:53:04.304 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: order 22 (4 MiB objects) 2023-10-11T10:53:04.304 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: snapshot_count: 20 2023-10-11T10:53:04.304 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: id: 10afefb6b972 2023-10-11T10:53:04.305 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: block_name_prefix: rbd_data.10afefb6b972 2023-10-11T10:53:04.305 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: format: 2 2023-10-11T10:53:04.305 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-10-11T10:53:04.305 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: op_features: 2023-10-11T10:53:04.305 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: flags: 2023-10-11T10:53:04.306 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: create_timestamp: Wed Oct 11 10:44:53 2023 2023-10-11T10:53:04.306 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: access_timestamp: Wed Oct 11 10:51:27 2023 2023-10-11T10:53:04.306 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: modify_timestamp: Wed Oct 11 10:44:53 2023 2023-10-11T10:53:04.306 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: protected: False 2023-10-11T10:53:04.306 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring state: enabled 2023-10-11T10:53:04.307 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring mode: snapshot 2023-10-11T10:53:04.307 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring global id: 3dd93db4-c9cc-462c-8b49-007eba9a90d2 2023-10-11T10:53:04.307 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring primary: false 2023-10-11T10:53:04.308 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T10:53:04.308 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n snap9-clean ']' 2023-10-11T10:53:04.309 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ compare_image_snaps mirror test snap9-clean 2023-10-11T10:53:04.309 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:53:04.309 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:53:04.309 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap9-clean 2023-10-11T10:53:04.309 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local ret=0 2023-10-11T10:53:04.309 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local rmt_export=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export 2023-10-11T10:53:04.310 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local loc_export=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:53:04.310 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rm -f /tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:53:04.310 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 -p mirror export test@snap9-clean /tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export 2023-10-11T10:53:04.897 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-10-11T10:53:04.907 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 -p mirror export test@snap9-clean /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:53:05.647 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-10-11T10:53:05.657 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ cmp /tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:53:05.873 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rm -f /tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:53:06.013 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T10:53:06.013 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ compare_image_snaps mirror test snap10 2023-10-11T10:53:06.013 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:53:06.014 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:53:06.015 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap10 2023-10-11T10:53:06.015 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local ret=0 2023-10-11T10:53:06.015 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local rmt_export=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export 2023-10-11T10:53:06.015 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local loc_export=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:53:06.015 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rm -f /tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:53:06.016 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 -p mirror export test@snap10 /tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export 2023-10-11T10:53:06.581 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-10-11T10:53:06.589 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 -p mirror export test@snap10 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:53:07.352 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-10-11T10:53:07.362 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ cmp /tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:53:07.579 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rm -f /tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:53:07.720 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T10:53:07.720 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ clean_snap_name=snap10-clean 2023-10-11T10:53:07.720 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ create_snap cluster2 mirror test snap10-clean 2023-10-11T10:53:07.720 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:53:07.721 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:53:07.721 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:53:07.721 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap10-clean 2023-10-11T10:53:07.721 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap10-clean --debug-rbd=20 --debug-journaler=20 2023-10-11T10:53:07.921 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_replay_started cluster1 mirror test 2023-10-11T10:53:07.921 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T10:53:07.921 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:53:07.921 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:53:07.921 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2023-10-11T10:53:07.922 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T10:53:07.922 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:53:07.922 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:53:07.922 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local state=started 2023-10-11T10:53:07.922 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T10:53:07.923 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-10-11T10:53:07.923 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 1 2023-10-11T10:53:08.922 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test_image_replay_state cluster1 mirror test started 2023-10-11T10:53:08.922 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T10:53:08.922 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:53:08.922 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:53:08.923 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local test_state=started 2023-10-11T10:53:08.923 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local status_result 2023-10-11T10:53:08.923 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_state=stopped 2023-10-11T10:53:08.924 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2023-10-11T10:53:08.924 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ grep -i state 2023-10-11T10:53:08.924 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_instance=cluster1 2023-10-11T10:53:08.925 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T10:53:08.925 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T10:53:08.925 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance=cluster1 2023-10-11T10:53:08.925 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local loop_instance 2023-10-11T10:53:08.925 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-10-11T10:53:08.926 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sleep 0 2023-10-11T10:53:08.926 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-10-11T10:53:08.926 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ seq 0 3 2023-10-11T10:53:08.928 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-10-11T10:53:08.928 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-10-11T10:53:08.928 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1:0 2023-10-11T10:53:08.928 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T10:53:08.928 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance 2023-10-11T10:53:08.929 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-10-11T10:53:08.929 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local val=cluster1:0 2023-10-11T10:53:08.929 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_var_name=cluster 2023-10-11T10:53:08.929 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance_var_name=instance 2023-10-11T10:53:08.929 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ cluster=cluster1 2023-10-11T10:53:08.930 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ instance=0 2023-10-11T10:53:08.930 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' 0 = cluster1:0 ']' 2023-10-11T10:53:08.930 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval cluster=cluster1 2023-10-11T10:53:08.930 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T10:53:08.930 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval instance=0 2023-10-11T10:53:08.930 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=0 2023-10-11T10:53:08.931 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-10-11T10:53:08.931 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local local_cluster=cluster1:0 2023-10-11T10:53:08.931 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster=cluster1 2023-10-11T10:53:08.931 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance 2023-10-11T10:53:08.931 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-10-11T10:53:08.932 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local val=cluster1:0 2023-10-11T10:53:08.933 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster_var_name=local_cluster 2023-10-11T10:53:08.933 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance_var_name=instance 2023-10-11T10:53:08.933 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T10:53:08.933 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=0 2023-10-11T10:53:08.933 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ '[' 0 = cluster1:0 ']' 2023-10-11T10:53:08.933 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval local_cluster=cluster1 2023-10-11T10:53:08.934 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ local_cluster=cluster1 2023-10-11T10:53:08.934 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval instance=0 2023-10-11T10:53:08.934 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ instance=0 2023-10-11T10:53:08.934 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-10-11T10:53:08.948 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T10:53:08.949 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T10:53:08.949 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T10:53:08.949 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-10-11T10:53:09.049 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-10-11T10:53:09.049 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd mirror status : get status for rbd mirror 2023-10-11T10:53:09.049 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:admin_socket: invalid command 2023-10-11T10:53:09.052 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-10-11T10:53:09.052 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-10-11T10:53:09.052 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1:1 2023-10-11T10:53:09.052 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T10:53:09.053 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance 2023-10-11T10:53:09.053 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-10-11T10:53:09.053 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local val=cluster1:1 2023-10-11T10:53:09.053 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_var_name=cluster 2023-10-11T10:53:09.053 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance_var_name=instance 2023-10-11T10:53:09.053 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ cluster=cluster1 2023-10-11T10:53:09.054 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ instance=1 2023-10-11T10:53:09.054 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' 1 = cluster1:1 ']' 2023-10-11T10:53:09.054 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval cluster=cluster1 2023-10-11T10:53:09.054 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T10:53:09.054 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval instance=1 2023-10-11T10:53:09.055 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=1 2023-10-11T10:53:09.055 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-10-11T10:53:09.055 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local local_cluster=cluster1:1 2023-10-11T10:53:09.055 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster=cluster1 2023-10-11T10:53:09.055 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance 2023-10-11T10:53:09.055 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-10-11T10:53:09.056 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local val=cluster1:1 2023-10-11T10:53:09.056 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster_var_name=local_cluster 2023-10-11T10:53:09.056 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance_var_name=instance 2023-10-11T10:53:09.056 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T10:53:09.056 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=1 2023-10-11T10:53:09.056 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ '[' 1 = cluster1:1 ']' 2023-10-11T10:53:09.057 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval local_cluster=cluster1 2023-10-11T10:53:09.057 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ local_cluster=cluster1 2023-10-11T10:53:09.057 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval instance=1 2023-10-11T10:53:09.057 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ instance=1 2023-10-11T10:53:09.057 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-10-11T10:53:09.071 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T10:53:09.071 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T10:53:09.071 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T10:53:09.071 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-10-11T10:53:09.175 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ return 0 2023-10-11T10:53:09.175 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ status_result=' "state": "Replaying"' 2023-10-11T10:53:09.176 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ echo ' "state": "Replaying"' 2023-10-11T10:53:09.176 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ grep -i Replaying 2023-10-11T10:53:09.176 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: "state": "Replaying" 2023-10-11T10:53:09.178 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ current_state=started 2023-10-11T10:53:09.178 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test started = started 2023-10-11T10:53:09.178 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T10:53:09.178 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2023-10-11T10:53:09.179 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local local_cluster=cluster1 2023-10-11T10:53:09.179 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:53:09.179 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:53:09.179 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:53:09.179 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' snapshot = journal ']' 2023-10-11T10:53:09.179 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' snapshot = snapshot ']' 2023-10-11T10:53:09.180 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2023-10-11T10:53:09.180 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local local_cluster=cluster1 2023-10-11T10:53:09.180 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:53:09.180 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:53:09.180 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:53:09.180 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ mkfname cluster2-mirror-test.status 2023-10-11T10:53:09.181 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ echo cluster2-mirror-test.status 2023-10-11T10:53:09.181 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -e 's|[/ ]|_|g' 2023-10-11T10:53:09.182 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local status_log=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.status 2023-10-11T10:53:09.182 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ mkfname cluster1-mirror-test.status 2023-10-11T10:53:09.182 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ echo cluster1-mirror-test.status 2023-10-11T10:53:09.182 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -e 's|[/ ]|_|g' 2023-10-11T10:53:09.183 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local local_status_log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.status 2023-10-11T10:53:09.183 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ mirror_image_snapshot cluster2 mirror test 2023-10-11T10:53:09.183 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:53:09.183 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:53:09.183 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:53:09.183 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2023-10-11T10:53:10.910 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:Snapshot ID: 44 2023-10-11T10:53:10.938 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.status 2023-10-11T10:53:10.939 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:53:10.939 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:53:10.939 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:53:10.939 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.status 2023-10-11T10:53:10.939 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2023-10-11T10:53:10.939 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T10:53:11.005 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-10-11T10:53:11.007 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snapshot_id=44 2023-10-11T10:53:11.008 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T10:53:11.008 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T10:53:11.008 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.2 2023-10-11T10:53:11.209 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.status 2023-10-11T10:53:11.209 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T10:53:11.209 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:53:11.210 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:53:11.210 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.status 2023-10-11T10:53:11.210 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-10-11T10:53:11.210 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T10:53:11.257 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T10:53:11.258 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T10:53:11.259 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:-:1.1: Document is empty 2023-10-11T10:53:11.259 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: 2023-10-11T10:53:11.259 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:^ 2023-10-11T10:53:11.260 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id= 2023-10-11T10:53:11.260 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 44 = '' 2023-10-11T10:53:11.260 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T10:53:11.260 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.4 2023-10-11T10:53:11.660 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.status 2023-10-11T10:53:11.661 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T10:53:11.661 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:53:11.661 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:53:11.661 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.status 2023-10-11T10:53:11.661 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-10-11T10:53:11.662 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T10:53:11.717 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T10:53:11.719 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id=44 2023-10-11T10:53:11.719 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 44 = 44 2023-10-11T10:53:11.719 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T10:53:11.719 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_snap_present cluster1 mirror test snap10-clean 2023-10-11T10:53:11.720 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T10:53:11.720 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:53:11.720 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:53:11.720 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap10-clean 2023-10-11T10:53:11.720 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T10:53:11.721 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-10-11T10:53:11.721 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 1 2023-10-11T10:53:12.719 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 info mirror/test@snap10-clean 2023-10-11T10:53:12.773 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:rbd image 'test': 2023-10-11T10:53:12.774 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: size 512 MiB in 128 objects 2023-10-11T10:53:12.774 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: order 22 (4 MiB objects) 2023-10-11T10:53:12.774 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: snapshot_count: 21 2023-10-11T10:53:12.774 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: id: 10afefb6b972 2023-10-11T10:53:12.774 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: block_name_prefix: rbd_data.10afefb6b972 2023-10-11T10:53:12.775 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: format: 2 2023-10-11T10:53:12.775 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-10-11T10:53:12.775 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: op_features: 2023-10-11T10:53:12.775 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: flags: 2023-10-11T10:53:12.775 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: create_timestamp: Wed Oct 11 10:44:53 2023 2023-10-11T10:53:12.776 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: access_timestamp: Wed Oct 11 10:53:04 2023 2023-10-11T10:53:12.776 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: modify_timestamp: Wed Oct 11 10:44:53 2023 2023-10-11T10:53:12.776 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: protected: False 2023-10-11T10:53:12.776 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring state: enabled 2023-10-11T10:53:12.776 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring mode: snapshot 2023-10-11T10:53:12.777 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring global id: 3dd93db4-c9cc-462c-8b49-007eba9a90d2 2023-10-11T10:53:12.777 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring primary: false 2023-10-11T10:53:12.777 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T10:53:12.777 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ seq 1 10 2023-10-11T10:53:12.779 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 10` 2023-10-11T10:53:12.779 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ snap_name=snap1 2023-10-11T10:53:12.779 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ compare_image_snaps mirror test snap1 2023-10-11T10:53:12.779 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:53:12.779 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:53:12.780 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap1 2023-10-11T10:53:12.780 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local ret=0 2023-10-11T10:53:12.780 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local rmt_export=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export 2023-10-11T10:53:12.780 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local loc_export=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:53:12.780 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rm -f /tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:53:12.781 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 -p mirror export test@snap1 /tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export 2023-10-11T10:53:13.528 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-10-11T10:53:13.536 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 -p mirror export test@snap1 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:53:14.262 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-10-11T10:53:14.271 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ cmp /tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:53:14.482 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rm -f /tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:53:14.623 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T10:53:14.623 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ snap_name=snap1-clean 2023-10-11T10:53:14.623 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ compare_image_snaps mirror test snap1-clean 2023-10-11T10:53:14.623 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:53:14.624 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:53:14.624 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap1-clean 2023-10-11T10:53:14.624 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local ret=0 2023-10-11T10:53:14.624 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local rmt_export=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export 2023-10-11T10:53:14.624 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local loc_export=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:53:14.625 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rm -f /tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:53:14.625 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 -p mirror export test@snap1-clean /tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export 2023-10-11T10:53:15.194 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-10-11T10:53:15.204 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 -p mirror export test@snap1-clean /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:53:15.894 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-10-11T10:53:15.904 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ cmp /tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:53:16.116 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rm -f /tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:53:16.257 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T10:53:16.258 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 10` 2023-10-11T10:53:16.258 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ snap_name=snap2 2023-10-11T10:53:16.258 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ compare_image_snaps mirror test snap2 2023-10-11T10:53:16.258 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:53:16.258 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:53:16.259 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap2 2023-10-11T10:53:16.259 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local ret=0 2023-10-11T10:53:16.259 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local rmt_export=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export 2023-10-11T10:53:16.259 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local loc_export=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:53:16.259 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rm -f /tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:53:16.259 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 -p mirror export test@snap2 /tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export 2023-10-11T10:53:16.843 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-10-11T10:53:16.852 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 -p mirror export test@snap2 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:53:17.514 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-10-11T10:53:17.523 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ cmp /tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:53:17.737 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rm -f /tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:53:17.879 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T10:53:17.880 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ snap_name=snap2-clean 2023-10-11T10:53:17.880 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ compare_image_snaps mirror test snap2-clean 2023-10-11T10:53:17.880 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:53:17.880 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:53:17.880 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap2-clean 2023-10-11T10:53:17.881 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local ret=0 2023-10-11T10:53:17.881 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local rmt_export=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export 2023-10-11T10:53:17.881 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local loc_export=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:53:17.881 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rm -f /tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:53:17.882 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 -p mirror export test@snap2-clean /tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export 2023-10-11T10:53:18.457 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-10-11T10:53:18.464 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 -p mirror export test@snap2-clean /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:53:19.184 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-10-11T10:53:19.193 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ cmp /tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:53:19.406 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rm -f /tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:53:19.523 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2023-10-11T10:53:19.523 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2023-10-11T10:53:19.524 INFO:tasks.rbd_mirror_thrash:waiting for 5 secs before reviving daemons 2023-10-11T10:53:19.524 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi111.stderr:2023-10-11T10:53:19.524+0000 7f2024ebb700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 109982) UID: 1000 2023-10-11T10:53:19.549 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T10:53:19.550 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 10` 2023-10-11T10:53:19.550 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ snap_name=snap3 2023-10-11T10:53:19.550 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ compare_image_snaps mirror test snap3 2023-10-11T10:53:19.550 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:53:19.550 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:53:19.551 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap3 2023-10-11T10:53:19.551 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local ret=0 2023-10-11T10:53:19.551 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local rmt_export=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export 2023-10-11T10:53:19.551 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local loc_export=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:53:19.551 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rm -f /tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:53:19.551 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 -p mirror export test@snap3 /tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export 2023-10-11T10:53:20.133 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-10-11T10:53:20.142 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 -p mirror export test@snap3 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:53:20.850 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-10-11T10:53:20.861 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ cmp /tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:53:21.073 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rm -f /tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:53:21.214 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T10:53:21.214 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ snap_name=snap3-clean 2023-10-11T10:53:21.214 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ compare_image_snaps mirror test snap3-clean 2023-10-11T10:53:21.215 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:53:21.215 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:53:21.215 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap3-clean 2023-10-11T10:53:21.215 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local ret=0 2023-10-11T10:53:21.216 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local rmt_export=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export 2023-10-11T10:53:21.216 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local loc_export=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:53:21.216 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rm -f /tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:53:21.216 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 -p mirror export test@snap3-clean /tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export 2023-10-11T10:53:21.782 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-10-11T10:53:21.791 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 -p mirror export test@snap3-clean /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:53:22.538 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-10-11T10:53:22.547 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ cmp /tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:53:22.762 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rm -f /tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:53:22.903 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T10:53:22.904 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 10` 2023-10-11T10:53:22.904 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ snap_name=snap4 2023-10-11T10:53:22.904 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ compare_image_snaps mirror test snap4 2023-10-11T10:53:22.904 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:53:22.904 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:53:22.905 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap4 2023-10-11T10:53:22.905 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local ret=0 2023-10-11T10:53:22.905 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local rmt_export=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export 2023-10-11T10:53:22.905 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local loc_export=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:53:22.905 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rm -f /tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:53:22.906 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 -p mirror export test@snap4 /tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export 2023-10-11T10:53:23.495 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-10-11T10:53:23.501 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 -p mirror export test@snap4 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:53:24.238 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-10-11T10:53:24.247 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ cmp /tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:53:24.463 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rm -f /tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:53:24.524 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2023-10-11T10:53:24.525 INFO:teuthology.orchestra.run:waiting for 600 2023-10-11T10:53:24.525 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2023-10-11T10:53:24.525 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2023-10-11T10:53:24.526 DEBUG:teuthology.orchestra.run.smithi111:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2023-10-11T10:53:24.528 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2023-10-11T10:53:24.528 INFO:tasks.rbd_mirror_thrash:waiting for 110 secs before thrashing 2023-10-11T10:53:24.606 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T10:53:24.606 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ snap_name=snap4-clean 2023-10-11T10:53:24.606 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ compare_image_snaps mirror test snap4-clean 2023-10-11T10:53:24.607 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:53:24.607 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:53:24.607 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap4-clean 2023-10-11T10:53:24.607 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local ret=0 2023-10-11T10:53:24.607 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local rmt_export=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export 2023-10-11T10:53:24.607 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local loc_export=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:53:24.608 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rm -f /tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:53:24.608 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 -p mirror export test@snap4-clean /tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export 2023-10-11T10:53:25.183 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-10-11T10:53:25.194 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 -p mirror export test@snap4-clean /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:53:25.899 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-10-11T10:53:25.907 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ cmp /tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:53:26.118 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rm -f /tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:53:26.260 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T10:53:26.260 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 10` 2023-10-11T10:53:26.260 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ snap_name=snap5 2023-10-11T10:53:26.261 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ compare_image_snaps mirror test snap5 2023-10-11T10:53:26.261 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:53:26.261 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:53:26.261 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap5 2023-10-11T10:53:26.261 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local ret=0 2023-10-11T10:53:26.262 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local rmt_export=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export 2023-10-11T10:53:26.262 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local loc_export=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:53:26.262 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rm -f /tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:53:26.262 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 -p mirror export test@snap5 /tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export 2023-10-11T10:53:26.855 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-10-11T10:53:26.863 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 -p mirror export test@snap5 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:53:27.592 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-10-11T10:53:27.604 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ cmp /tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:53:27.821 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rm -f /tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:53:27.965 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T10:53:27.965 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ snap_name=snap5-clean 2023-10-11T10:53:27.965 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ compare_image_snaps mirror test snap5-clean 2023-10-11T10:53:27.965 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:53:27.965 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:53:27.966 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap5-clean 2023-10-11T10:53:27.966 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local ret=0 2023-10-11T10:53:27.966 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local rmt_export=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export 2023-10-11T10:53:27.966 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local loc_export=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:53:27.966 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rm -f /tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:53:27.967 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 -p mirror export test@snap5-clean /tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export 2023-10-11T10:53:28.551 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-10-11T10:53:28.560 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 -p mirror export test@snap5-clean /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:53:29.295 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-10-11T10:53:29.307 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ cmp /tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:53:29.520 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rm -f /tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:53:29.663 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T10:53:29.664 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 10` 2023-10-11T10:53:29.664 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ snap_name=snap6 2023-10-11T10:53:29.664 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ compare_image_snaps mirror test snap6 2023-10-11T10:53:29.664 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:53:29.664 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:53:29.665 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap6 2023-10-11T10:53:29.665 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local ret=0 2023-10-11T10:53:29.665 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local rmt_export=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export 2023-10-11T10:53:29.665 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local loc_export=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:53:29.665 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rm -f /tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:53:29.666 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 -p mirror export test@snap6 /tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export 2023-10-11T10:53:30.239 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-10-11T10:53:30.251 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 -p mirror export test@snap6 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:53:31.004 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-10-11T10:53:31.012 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ cmp /tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:53:31.225 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rm -f /tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:53:31.366 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T10:53:31.366 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ snap_name=snap6-clean 2023-10-11T10:53:31.367 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ compare_image_snaps mirror test snap6-clean 2023-10-11T10:53:31.367 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:53:31.367 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:53:31.367 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap6-clean 2023-10-11T10:53:31.367 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local ret=0 2023-10-11T10:53:31.367 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local rmt_export=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export 2023-10-11T10:53:31.368 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local loc_export=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:53:31.368 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rm -f /tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:53:31.368 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 -p mirror export test@snap6-clean /tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export 2023-10-11T10:53:31.946 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-10-11T10:53:31.956 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 -p mirror export test@snap6-clean /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:53:32.642 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-10-11T10:53:32.650 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ cmp /tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:53:32.863 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rm -f /tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:53:33.006 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T10:53:33.006 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 10` 2023-10-11T10:53:33.006 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ snap_name=snap7 2023-10-11T10:53:33.006 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ compare_image_snaps mirror test snap7 2023-10-11T10:53:33.006 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:53:33.007 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:53:33.007 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap7 2023-10-11T10:53:33.007 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local ret=0 2023-10-11T10:53:33.007 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local rmt_export=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export 2023-10-11T10:53:33.007 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local loc_export=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:53:33.008 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rm -f /tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:53:33.008 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 -p mirror export test@snap7 /tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export 2023-10-11T10:53:33.588 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-10-11T10:53:33.597 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 -p mirror export test@snap7 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:53:34.266 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-10-11T10:53:34.275 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ cmp /tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:53:34.488 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rm -f /tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:53:34.629 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T10:53:34.629 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ snap_name=snap7-clean 2023-10-11T10:53:34.630 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ compare_image_snaps mirror test snap7-clean 2023-10-11T10:53:34.630 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:53:34.630 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:53:34.630 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap7-clean 2023-10-11T10:53:34.631 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local ret=0 2023-10-11T10:53:34.631 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local rmt_export=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export 2023-10-11T10:53:34.631 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local loc_export=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:53:34.631 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rm -f /tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:53:34.631 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 -p mirror export test@snap7-clean /tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export 2023-10-11T10:53:35.242 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-10-11T10:53:35.254 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 -p mirror export test@snap7-clean /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:53:35.987 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-10-11T10:53:35.995 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ cmp /tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:53:36.206 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rm -f /tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:53:36.348 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T10:53:36.349 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 10` 2023-10-11T10:53:36.349 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ snap_name=snap8 2023-10-11T10:53:36.349 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ compare_image_snaps mirror test snap8 2023-10-11T10:53:36.349 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:53:36.350 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:53:36.350 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap8 2023-10-11T10:53:36.350 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local ret=0 2023-10-11T10:53:36.350 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local rmt_export=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export 2023-10-11T10:53:36.350 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local loc_export=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:53:36.351 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rm -f /tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:53:36.351 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 -p mirror export test@snap8 /tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export 2023-10-11T10:53:36.923 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-10-11T10:53:36.930 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 -p mirror export test@snap8 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:53:37.656 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-10-11T10:53:37.666 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ cmp /tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:53:37.883 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rm -f /tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:53:38.027 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T10:53:38.027 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ snap_name=snap8-clean 2023-10-11T10:53:38.027 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ compare_image_snaps mirror test snap8-clean 2023-10-11T10:53:38.027 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:53:38.027 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:53:38.028 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap8-clean 2023-10-11T10:53:38.028 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local ret=0 2023-10-11T10:53:38.028 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local rmt_export=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export 2023-10-11T10:53:38.028 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local loc_export=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:53:38.028 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rm -f /tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:53:38.029 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 -p mirror export test@snap8-clean /tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export 2023-10-11T10:53:38.591 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-10-11T10:53:38.601 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 -p mirror export test@snap8-clean /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:53:39.317 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-10-11T10:53:39.325 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ cmp /tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:53:39.540 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rm -f /tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:53:39.684 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T10:53:39.684 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 10` 2023-10-11T10:53:39.684 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ snap_name=snap9 2023-10-11T10:53:39.684 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ compare_image_snaps mirror test snap9 2023-10-11T10:53:39.684 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:53:39.685 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:53:39.685 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap9 2023-10-11T10:53:39.685 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local ret=0 2023-10-11T10:53:39.685 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local rmt_export=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export 2023-10-11T10:53:39.685 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local loc_export=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:53:39.686 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rm -f /tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:53:39.686 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 -p mirror export test@snap9 /tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export 2023-10-11T10:53:40.288 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-10-11T10:53:40.298 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 -p mirror export test@snap9 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:53:40.902 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-10-11T10:53:40.911 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ cmp /tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:53:41.121 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rm -f /tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:53:41.263 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T10:53:41.263 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ snap_name=snap9-clean 2023-10-11T10:53:41.263 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ compare_image_snaps mirror test snap9-clean 2023-10-11T10:53:41.263 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:53:41.263 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:53:41.264 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap9-clean 2023-10-11T10:53:41.264 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local ret=0 2023-10-11T10:53:41.264 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local rmt_export=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export 2023-10-11T10:53:41.264 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local loc_export=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:53:41.264 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rm -f /tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:53:41.265 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 -p mirror export test@snap9-clean /tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export 2023-10-11T10:53:41.846 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-10-11T10:53:41.856 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 -p mirror export test@snap9-clean /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:53:42.579 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-10-11T10:53:42.588 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ cmp /tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:53:42.801 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rm -f /tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:53:42.946 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T10:53:42.946 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 10` 2023-10-11T10:53:42.947 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ snap_name=snap10 2023-10-11T10:53:42.947 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ compare_image_snaps mirror test snap10 2023-10-11T10:53:42.947 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:53:42.947 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:53:42.947 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap10 2023-10-11T10:53:42.947 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local ret=0 2023-10-11T10:53:42.948 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local rmt_export=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export 2023-10-11T10:53:42.948 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local loc_export=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:53:42.948 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rm -f /tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:53:42.949 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 -p mirror export test@snap10 /tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export 2023-10-11T10:53:43.525 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.4733499894953591) > weight (0.25) 2023-10-11T10:53:43.526 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.1 2023-10-11T10:53:43.526 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Sent signal 15 2023-10-11T10:53:43.526 INFO:tasks.rbd_mirror_thrash:waiting for 6 secs before reviving daemons 2023-10-11T10:53:43.527 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi111.stderr:2023-10-11T10:53:43.526+0000 7f01c0ccf700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 112354) UID: 1000 2023-10-11T10:53:43.539 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-10-11T10:53:43.550 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 -p mirror export test@snap10 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:53:44.262 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-10-11T10:53:44.272 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ cmp /tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:53:44.486 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rm -f /tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:53:44.630 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T10:53:44.631 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ snap_name=snap10-clean 2023-10-11T10:53:44.631 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ compare_image_snaps mirror test snap10-clean 2023-10-11T10:53:44.631 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:53:44.631 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:53:44.631 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap10-clean 2023-10-11T10:53:44.632 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local ret=0 2023-10-11T10:53:44.632 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local rmt_export=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export 2023-10-11T10:53:44.632 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local loc_export=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:53:44.632 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rm -f /tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:53:44.632 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 -p mirror export test@snap10-clean /tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export 2023-10-11T10:53:45.222 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-10-11T10:53:45.229 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 -p mirror export test@snap10-clean /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:53:45.996 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-10-11T10:53:46.006 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ cmp /tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:53:46.221 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rm -f /tmp/tmp.nGgfBEp0YO/cluster2-mirror-test.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-test.export 2023-10-11T10:53:46.367 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T10:53:46.368 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ seq 1 10 2023-10-11T10:53:46.368 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 10` 2023-10-11T10:53:46.368 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ snap_name=snap1 2023-10-11T10:53:46.368 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_snapshot cluster2 mirror test snap1 2023-10-11T10:53:46.369 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:53:46.369 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:53:46.369 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:53:46.369 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap=snap1 2023-10-11T10:53:46.370 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap1 2023-10-11T10:53:47.291 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Removing snap: 100% complete...done. 2023-10-11T10:53:47.314 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ snap_name=snap1-clean 2023-10-11T10:53:47.315 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_snapshot cluster2 mirror test snap1-clean 2023-10-11T10:53:47.315 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:53:47.315 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:53:47.315 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:53:47.316 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap=snap1-clean 2023-10-11T10:53:47.316 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap1-clean 2023-10-11T10:53:48.295 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Removing snap: 100% complete...done. 2023-10-11T10:53:48.330 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 10` 2023-10-11T10:53:48.331 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ snap_name=snap2 2023-10-11T10:53:48.331 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_snapshot cluster2 mirror test snap2 2023-10-11T10:53:48.331 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:53:48.331 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:53:48.331 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:53:48.332 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap=snap2 2023-10-11T10:53:48.332 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap2 2023-10-11T10:53:49.293 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Removing snap: 100% complete...done. 2023-10-11T10:53:49.319 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ snap_name=snap2-clean 2023-10-11T10:53:49.319 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_snapshot cluster2 mirror test snap2-clean 2023-10-11T10:53:49.319 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:53:49.320 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:53:49.320 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:53:49.320 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap=snap2-clean 2023-10-11T10:53:49.320 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap2-clean 2023-10-11T10:53:49.528 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.1 2023-10-11T10:53:49.528 INFO:teuthology.orchestra.run:waiting for 600 2023-10-11T10:53:49.528 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.1 2023-10-11T10:53:49.528 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2023-10-11T10:53:49.529 DEBUG:teuthology.orchestra.run.smithi111:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2023-10-11T10:53:49.532 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2023-10-11T10:53:49.532 INFO:tasks.rbd_mirror_thrash:waiting for 109 secs before thrashing 2023-10-11T10:53:50.303 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Removing snap: 100% complete...done. 2023-10-11T10:53:50.330 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 10` 2023-10-11T10:53:50.331 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ snap_name=snap3 2023-10-11T10:53:50.331 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_snapshot cluster2 mirror test snap3 2023-10-11T10:53:50.331 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:53:50.331 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:53:50.332 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:53:50.332 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap=snap3 2023-10-11T10:53:50.332 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap3 2023-10-11T10:53:51.308 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Removing snap: 100% complete...done. 2023-10-11T10:53:51.340 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ snap_name=snap3-clean 2023-10-11T10:53:51.340 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_snapshot cluster2 mirror test snap3-clean 2023-10-11T10:53:51.340 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:53:51.341 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:53:51.341 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:53:51.341 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap=snap3-clean 2023-10-11T10:53:51.341 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap3-clean 2023-10-11T10:53:52.316 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Removing snap: 100% complete...done. 2023-10-11T10:53:52.359 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 10` 2023-10-11T10:53:52.359 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ snap_name=snap4 2023-10-11T10:53:52.359 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_snapshot cluster2 mirror test snap4 2023-10-11T10:53:52.359 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:53:52.359 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:53:52.360 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:53:52.360 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap=snap4 2023-10-11T10:53:52.360 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap4 2023-10-11T10:53:53.314 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Removing snap: 100% complete...done. 2023-10-11T10:53:53.338 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ snap_name=snap4-clean 2023-10-11T10:53:53.338 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_snapshot cluster2 mirror test snap4-clean 2023-10-11T10:53:53.338 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:53:53.338 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:53:53.338 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:53:53.339 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap=snap4-clean 2023-10-11T10:53:53.339 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap4-clean 2023-10-11T10:53:54.330 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Removing snap: 100% complete...done. 2023-10-11T10:53:54.365 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 10` 2023-10-11T10:53:54.365 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ snap_name=snap5 2023-10-11T10:53:54.365 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_snapshot cluster2 mirror test snap5 2023-10-11T10:53:54.365 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:53:54.366 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:53:54.366 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:53:54.366 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap=snap5 2023-10-11T10:53:54.366 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap5 2023-10-11T10:53:54.958 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Removing snap: 100% complete...done. 2023-10-11T10:53:54.981 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ snap_name=snap5-clean 2023-10-11T10:53:54.981 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_snapshot cluster2 mirror test snap5-clean 2023-10-11T10:53:54.982 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:53:54.982 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:53:54.982 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:53:54.983 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap=snap5-clean 2023-10-11T10:53:54.983 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap5-clean 2023-10-11T10:53:55.958 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Removing snap: 100% complete...done. 2023-10-11T10:53:55.993 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 10` 2023-10-11T10:53:55.994 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ snap_name=snap6 2023-10-11T10:53:55.994 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_snapshot cluster2 mirror test snap6 2023-10-11T10:53:55.994 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:53:55.994 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:53:55.994 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:53:55.995 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap=snap6 2023-10-11T10:53:55.995 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap6 2023-10-11T10:53:56.957 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Removing snap: 100% complete...done. 2023-10-11T10:53:56.982 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ snap_name=snap6-clean 2023-10-11T10:53:56.983 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_snapshot cluster2 mirror test snap6-clean 2023-10-11T10:53:56.983 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:53:56.983 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:53:56.983 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:53:56.983 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap=snap6-clean 2023-10-11T10:53:56.984 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap6-clean 2023-10-11T10:53:57.968 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Removing snap: 100% complete...done. 2023-10-11T10:53:57.995 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 10` 2023-10-11T10:53:57.995 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ snap_name=snap7 2023-10-11T10:53:57.996 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_snapshot cluster2 mirror test snap7 2023-10-11T10:53:57.996 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:53:57.996 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:53:57.996 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:53:57.996 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap=snap7 2023-10-11T10:53:57.997 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap7 2023-10-11T10:53:58.963 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Removing snap: 100% complete...done. 2023-10-11T10:53:58.983 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ snap_name=snap7-clean 2023-10-11T10:53:58.983 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_snapshot cluster2 mirror test snap7-clean 2023-10-11T10:53:58.983 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:53:58.983 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:53:58.984 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:53:58.984 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap=snap7-clean 2023-10-11T10:53:58.984 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap7-clean 2023-10-11T10:53:59.961 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Removing snap: 100% complete...done. 2023-10-11T10:53:59.997 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 10` 2023-10-11T10:53:59.997 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ snap_name=snap8 2023-10-11T10:53:59.997 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_snapshot cluster2 mirror test snap8 2023-10-11T10:53:59.998 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:53:59.998 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:53:59.998 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:53:59.998 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap=snap8 2023-10-11T10:53:59.998 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap8 2023-10-11T10:54:00.981 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Removing snap: 100% complete...done. 2023-10-11T10:54:01.005 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ snap_name=snap8-clean 2023-10-11T10:54:01.006 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_snapshot cluster2 mirror test snap8-clean 2023-10-11T10:54:01.006 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:54:01.006 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:54:01.006 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:54:01.006 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap=snap8-clean 2023-10-11T10:54:01.007 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap8-clean 2023-10-11T10:54:01.971 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Removing snap: 100% complete...done. 2023-10-11T10:54:02.006 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 10` 2023-10-11T10:54:02.006 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ snap_name=snap9 2023-10-11T10:54:02.006 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_snapshot cluster2 mirror test snap9 2023-10-11T10:54:02.006 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:54:02.007 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:54:02.007 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:54:02.007 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap=snap9 2023-10-11T10:54:02.007 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap9 2023-10-11T10:54:02.982 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Removing snap: 100% complete...done. 2023-10-11T10:54:03.006 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ snap_name=snap9-clean 2023-10-11T10:54:03.006 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_snapshot cluster2 mirror test snap9-clean 2023-10-11T10:54:03.006 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:54:03.006 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:54:03.007 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:54:03.007 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap=snap9-clean 2023-10-11T10:54:03.007 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap9-clean 2023-10-11T10:54:03.992 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Removing snap: 100% complete...done. 2023-10-11T10:54:04.022 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 10` 2023-10-11T10:54:04.022 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ snap_name=snap10 2023-10-11T10:54:04.022 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_snapshot cluster2 mirror test snap10 2023-10-11T10:54:04.023 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:54:04.023 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:54:04.023 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:54:04.023 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap=snap10 2023-10-11T10:54:04.023 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap10 2023-10-11T10:54:04.964 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Removing snap: 100% complete...done. 2023-10-11T10:54:04.988 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ snap_name=snap10-clean 2023-10-11T10:54:04.988 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_snapshot cluster2 mirror test snap10-clean 2023-10-11T10:54:04.988 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:54:04.988 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:54:04.989 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:54:04.989 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap=snap10-clean 2023-10-11T10:54:04.989 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap10-clean 2023-10-11T10:54:06.004 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Removing snap: 100% complete...done. 2023-10-11T10:54:06.026 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_image_retry cluster2 mirror test 2023-10-11T10:54:06.026 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:54:06.026 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:54:06.027 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:54:06.027 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0 1 2 4 8 16 32 2023-10-11T10:54:06.027 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0 2023-10-11T10:54:06.028 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_image cluster2 mirror test 2023-10-11T10:54:06.028 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:54:06.028 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:54:06.028 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:54:06.028 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 snap purge mirror/test 2023-10-11T10:54:06.072 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 rm mirror/test 2023-10-11T10:54:06.187 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi111.stderr:2023-10-11T10:54:06.187+0000 7f277aaf2700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x556e9a1c4b60 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-10-11T10:54:06.188 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi111.stderr:2023-10-11T10:54:06.187+0000 7f2781b00700 -1 rbd::mirror::ImageReplayer: 0x556e998d0280 [3/3dd93db4-c9cc-462c-8b49-007eba9a90d2] operator(): start failed: (85) Interrupted system call should be restarted 2023-10-11T10:54:06.188 INFO:tasks.ceph.cluster1.osd.2.smithi050.stderr:2023-10-11T10:54:06.187+0000 7f3d93d55700 -1 reset not still connected to 0x55b71225d040 2023-10-11T10:54:06.188 INFO:tasks.ceph.cluster1.osd.2.smithi050.stderr:2023-10-11T10:54:06.187+0000 7f3d93d55700 -1 reset not still connected to 0x55b719d0dee0 2023-10-11T10:54:08.213 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2023-10-11T10:54:08.217 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T10:54:08.217 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_present cluster1 mirror test deleted 2023-10-11T10:54:08.217 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T10:54:08.218 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:54:08.218 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:54:08.218 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local state=deleted 2023-10-11T10:54:08.218 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image_id= 2023-10-11T10:54:08.219 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T10:54:08.219 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test -n '' 2023-10-11T10:54:08.219 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ get_image_id cluster1 mirror test 2023-10-11T10:54:08.219 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T10:54:08.219 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local pool=mirror 2023-10-11T10:54:08.220 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local image=test 2023-10-11T10:54:08.220 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ rbd --cluster=cluster1 info mirror/test 2023-10-11T10:54:08.220 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-10-11T10:54:08.249 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd: error opening image test: (2) No such file or directory 2023-10-11T10:54:08.252 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image_id= 2023-10-11T10:54:08.252 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-10-11T10:54:08.252 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.1 2023-10-11T10:54:08.354 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test_image_present cluster1 mirror test deleted '' 2023-10-11T10:54:08.355 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T10:54:08.355 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:54:08.355 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=test 2023-10-11T10:54:08.355 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local test_state=deleted 2023-10-11T10:54:08.356 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image_id= 2023-10-11T10:54:08.356 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_state=deleted 2023-10-11T10:54:08.356 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_image_id 2023-10-11T10:54:08.356 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ get_image_id cluster1 mirror test 2023-10-11T10:54:08.357 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T10:54:08.357 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local pool=mirror 2023-10-11T10:54:08.357 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local image=test 2023-10-11T10:54:08.357 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ rbd --cluster=cluster1 info mirror/test 2023-10-11T10:54:08.358 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-10-11T10:54:08.386 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd: error opening image test: (2) No such file or directory 2023-10-11T10:54:08.390 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ current_image_id= 2023-10-11T10:54:08.390 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test -n '' 2023-10-11T10:54:08.390 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test deleted = deleted 2023-10-11T10:54:08.391 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T10:54:08.391 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ testlog 'TEST: create many images' 2023-10-11T10:54:08.391 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ tee -a /tmp/tmp.nGgfBEp0YO/rbd-mirror.test.log 2023-10-11T10:54:08.391 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ date '+%F %T' 2023-10-11T10:54:08.394 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ echo 2023-10-11 10:54:08 TEST: create many images 2023-10-11T10:54:08.394 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:2023-10-11 10:54:08 TEST: create many images 2023-10-11T10:54:08.395 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ snap_name=snap 2023-10-11T10:54:08.395 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ seq 1 50 2023-10-11T10:54:08.395 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T10:54:08.395 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_1 2023-10-11T10:54:08.396 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ create_image_and_enable_mirror cluster2 mirror image_1 snapshot 128M 2023-10-11T10:54:08.396 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:54:08.396 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:54:08.396 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:54:08.397 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:54:08.397 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_1 2023-10-11T10:54:08.397 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:54:08.397 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local mode=snapshot 2023-10-11T10:54:08.398 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n snapshot ']' 2023-10-11T10:54:08.398 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:54:08.398 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ create_image cluster2 mirror image_1 128M 2023-10-11T10:54:08.399 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:54:08.399 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:54:08.399 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:54:08.399 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:54:08.400 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_1 2023-10-11T10:54:08.400 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:54:08.400 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local size=128 2023-10-11T10:54:08.400 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n 128M ']' 2023-10-11T10:54:08.401 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ size=128M 2023-10-11T10:54:08.401 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:54:08.401 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_1 2023-10-11T10:54:08.438 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' image = image ']' 2023-10-11T10:54:08.438 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ enable_mirror cluster2 mirror image_1 snapshot 2023-10-11T10:54:08.439 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:54:08.439 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:54:08.439 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_1 2023-10-11T10:54:08.439 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local mode=snapshot 2023-10-11T10:54:08.440 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_1 snapshot 2023-10-11T10:54:09.021 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:Mirroring enabled 2023-10-11T10:54:09.033 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 info mirror/image_1 2023-10-11T10:54:09.071 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:rbd image 'image_1': 2023-10-11T10:54:09.071 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: size 128 MiB in 32 objects 2023-10-11T10:54:09.071 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: order 22 (4 MiB objects) 2023-10-11T10:54:09.071 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: snapshot_count: 1 2023-10-11T10:54:09.072 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: id: 121ae3d24c92 2023-10-11T10:54:09.072 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: block_name_prefix: rbd_data.121ae3d24c92 2023-10-11T10:54:09.072 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: format: 2 2023-10-11T10:54:09.072 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-10-11T10:54:09.072 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: op_features: 2023-10-11T10:54:09.073 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: flags: 2023-10-11T10:54:09.073 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: create_timestamp: Wed Oct 11 10:54:08 2023 2023-10-11T10:54:09.073 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: access_timestamp: Wed Oct 11 10:54:08 2023 2023-10-11T10:54:09.073 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: modify_timestamp: Wed Oct 11 10:54:08 2023 2023-10-11T10:54:09.073 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring state: enabled 2023-10-11T10:54:09.073 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring mode: snapshot 2023-10-11T10:54:09.074 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring global id: eeb747e2-80b4-485a-95e6-a07d67abb191 2023-10-11T10:54:09.074 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring primary: true 2023-10-11T10:54:09.074 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n '' ']' 2023-10-11T10:54:09.075 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ stress_write_image cluster2 mirror image_1 2023-10-11T10:54:09.075 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:54:09.075 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:54:09.075 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_1 2023-10-11T10:54:09.075 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-10-11T10:54:09.076 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local duration=9 2023-10-11T10:54:09.076 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ set +e 2023-10-11T10:54:09.076 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_1 --debug-rbd=20 --debug-journaler=20 2023-10-11T10:54:09.273 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:100 IOs, 19931136 bytes 2023-10-11T10:54:09.425 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:200 IOs, 41950208 bytes 2023-10-11T10:54:09.687 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:300 IOs, 117135360 bytes 2023-10-11T10:54:09.849 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:400 IOs, 164907008 bytes 2023-10-11T10:54:10.033 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:500 IOs, 208036864 bytes 2023-10-11T10:54:10.204 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:600 IOs, 258701312 bytes 2023-10-11T10:54:10.385 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:700 IOs, 316628992 bytes 2023-10-11T10:54:10.563 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:800 IOs, 367265792 bytes 2023-10-11T10:54:10.676 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:900 IOs, 381900800 bytes 2023-10-11T10:54:10.820 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1000 IOs, 419203072 bytes 2023-10-11T10:54:10.931 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1100 IOs, 433271808 bytes 2023-10-11T10:54:11.050 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1200 IOs, 464138240 bytes 2023-10-11T10:54:11.219 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1300 IOs, 495467520 bytes 2023-10-11T10:54:11.427 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1400 IOs, 555824128 bytes 2023-10-11T10:54:11.556 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1500 IOs, 579767296 bytes 2023-10-11T10:54:11.683 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1600 IOs, 603014144 bytes 2023-10-11T10:54:11.881 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1700 IOs, 664206336 bytes 2023-10-11T10:54:11.986 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1800 IOs, 673776640 bytes 2023-10-11T10:54:12.112 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1900 IOs, 709413888 bytes 2023-10-11T10:54:12.366 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2000 IOs, 786205696 bytes 2023-10-11T10:54:12.546 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2100 IOs, 837403648 bytes 2023-10-11T10:54:12.653 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2200 IOs, 847510528 bytes 2023-10-11T10:54:12.737 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2300 IOs, 849090560 bytes 2023-10-11T10:54:12.955 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2400 IOs, 940037120 bytes 2023-10-11T10:54:13.145 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2500 IOs, 978227200 bytes 2023-10-11T10:54:13.369 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2600 IOs, 1067245568 bytes 2023-10-11T10:54:13.489 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2700 IOs, 1080314880 bytes 2023-10-11T10:54:13.630 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2800 IOs, 1105010688 bytes 2023-10-11T10:54:13.833 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2900 IOs, 1159000064 bytes 2023-10-11T10:54:14.049 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3000 IOs, 1228088320 bytes 2023-10-11T10:54:14.201 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3100 IOs, 1268275200 bytes 2023-10-11T10:54:14.348 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3200 IOs, 1285947392 bytes 2023-10-11T10:54:14.492 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3300 IOs, 1317101568 bytes 2023-10-11T10:54:14.735 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3400 IOs, 1403531264 bytes 2023-10-11T10:54:14.938 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3500 IOs, 1471142912 bytes 2023-10-11T10:54:15.150 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3600 IOs, 1530740736 bytes 2023-10-11T10:54:15.403 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3700 IOs, 1618627584 bytes 2023-10-11T10:54:15.541 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3800 IOs, 1652547584 bytes 2023-10-11T10:54:15.640 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3900 IOs, 1659531264 bytes 2023-10-11T10:54:15.737 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4000 IOs, 1660133376 bytes 2023-10-11T10:54:15.891 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4100 IOs, 1693918208 bytes 2023-10-11T10:54:15.981 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4200 IOs, 1701394432 bytes 2023-10-11T10:54:16.117 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4300 IOs, 1742683136 bytes 2023-10-11T10:54:16.410 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4400 IOs, 1823091712 bytes 2023-10-11T10:54:16.537 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4500 IOs, 1847657472 bytes 2023-10-11T10:54:16.666 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4600 IOs, 1868083200 bytes 2023-10-11T10:54:16.960 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4700 IOs, 1970719744 bytes 2023-10-11T10:54:17.306 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4800 IOs, 2003081216 bytes 2023-10-11T10:54:17.419 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4900 IOs, 2021195776 bytes 2023-10-11T10:54:17.577 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5000 IOs, 2060384256 bytes 2023-10-11T10:54:17.712 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5100 IOs, 2081126400 bytes 2023-10-11T10:54:17.835 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5200 IOs, 2096065536 bytes 2023-10-11T10:54:17.948 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5300 IOs, 2106115072 bytes 2023-10-11T10:54:18.089 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ error_code=124 2023-10-11T10:54:18.089 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ set -e 2023-10-11T10:54:18.089 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' 124 -eq 124 ']' 2023-10-11T10:54:18.090 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T10:54:18.090 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T10:54:18.090 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_2 2023-10-11T10:54:18.090 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ create_image_and_enable_mirror cluster2 mirror image_2 snapshot 128M 2023-10-11T10:54:18.091 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:54:18.091 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:54:18.091 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:54:18.092 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:54:18.092 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_2 2023-10-11T10:54:18.092 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:54:18.092 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local mode=snapshot 2023-10-11T10:54:18.093 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n snapshot ']' 2023-10-11T10:54:18.093 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:54:18.093 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ create_image cluster2 mirror image_2 128M 2023-10-11T10:54:18.093 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:54:18.094 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:54:18.094 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:54:18.094 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:54:18.094 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_2 2023-10-11T10:54:18.095 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:54:18.095 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local size=128 2023-10-11T10:54:18.095 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n 128M ']' 2023-10-11T10:54:18.096 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ size=128M 2023-10-11T10:54:18.096 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:54:18.096 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_2 2023-10-11T10:54:18.137 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' image = image ']' 2023-10-11T10:54:18.137 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ enable_mirror cluster2 mirror image_2 snapshot 2023-10-11T10:54:18.137 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:54:18.138 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:54:18.138 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_2 2023-10-11T10:54:18.138 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local mode=snapshot 2023-10-11T10:54:18.138 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_2 snapshot 2023-10-11T10:54:18.340 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:Mirroring enabled 2023-10-11T10:54:18.352 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 info mirror/image_2 2023-10-11T10:54:18.387 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:rbd image 'image_2': 2023-10-11T10:54:18.388 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: size 128 MiB in 32 objects 2023-10-11T10:54:18.388 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: order 22 (4 MiB objects) 2023-10-11T10:54:18.388 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: snapshot_count: 1 2023-10-11T10:54:18.389 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: id: 122462fcbbc4 2023-10-11T10:54:18.389 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: block_name_prefix: rbd_data.122462fcbbc4 2023-10-11T10:54:18.389 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: format: 2 2023-10-11T10:54:18.389 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-10-11T10:54:18.390 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: op_features: 2023-10-11T10:54:18.390 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: flags: 2023-10-11T10:54:18.390 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: create_timestamp: Wed Oct 11 10:54:18 2023 2023-10-11T10:54:18.391 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: access_timestamp: Wed Oct 11 10:54:18 2023 2023-10-11T10:54:18.391 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: modify_timestamp: Wed Oct 11 10:54:18 2023 2023-10-11T10:54:18.391 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring state: enabled 2023-10-11T10:54:18.391 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring mode: snapshot 2023-10-11T10:54:18.392 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring global id: 115d7fef-7d84-4b5a-aaf4-a5ee31ecb254 2023-10-11T10:54:18.392 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring primary: true 2023-10-11T10:54:18.393 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n '' ']' 2023-10-11T10:54:18.394 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ stress_write_image cluster2 mirror image_2 2023-10-11T10:54:18.394 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:54:18.394 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:54:18.394 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_2 2023-10-11T10:54:18.395 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-10-11T10:54:18.395 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local duration=12 2023-10-11T10:54:18.395 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ set +e 2023-10-11T10:54:18.396 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_2 --debug-rbd=20 --debug-journaler=20 2023-10-11T10:54:18.552 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:100 IOs, 11383808 bytes 2023-10-11T10:54:18.811 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:200 IOs, 92838912 bytes 2023-10-11T10:54:18.946 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:300 IOs, 116200448 bytes 2023-10-11T10:54:19.079 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:400 IOs, 141121536 bytes 2023-10-11T10:54:19.251 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:500 IOs, 192405504 bytes 2023-10-11T10:54:19.334 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:600 IOs, 195822592 bytes 2023-10-11T10:54:19.454 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:700 IOs, 218567680 bytes 2023-10-11T10:54:19.597 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:800 IOs, 266685440 bytes 2023-10-11T10:54:19.815 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:900 IOs, 324980736 bytes 2023-10-11T10:54:19.960 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1000 IOs, 350395392 bytes 2023-10-11T10:54:20.240 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1100 IOs, 452987904 bytes 2023-10-11T10:54:20.403 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1200 IOs, 484328448 bytes 2023-10-11T10:54:20.558 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1300 IOs, 525472768 bytes 2023-10-11T10:54:20.743 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1400 IOs, 577019904 bytes 2023-10-11T10:54:20.827 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1500 IOs, 577617920 bytes 2023-10-11T10:54:21.005 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1600 IOs, 635230208 bytes 2023-10-11T10:54:21.193 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1700 IOs, 726479872 bytes 2023-10-11T10:54:21.368 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1800 IOs, 760088576 bytes 2023-10-11T10:54:21.565 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1900 IOs, 819817472 bytes 2023-10-11T10:54:21.705 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2000 IOs, 855956480 bytes 2023-10-11T10:54:21.879 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2100 IOs, 898323456 bytes 2023-10-11T10:54:22.026 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2200 IOs, 929410048 bytes 2023-10-11T10:54:22.281 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2300 IOs, 1006218240 bytes 2023-10-11T10:54:22.576 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2400 IOs, 1114244096 bytes 2023-10-11T10:54:22.712 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2500 IOs, 1147064320 bytes 2023-10-11T10:54:22.839 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2600 IOs, 1174252544 bytes 2023-10-11T10:54:23.021 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2700 IOs, 1229619200 bytes 2023-10-11T10:54:23.152 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2800 IOs, 1261617152 bytes 2023-10-11T10:54:23.245 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2900 IOs, 1262235648 bytes 2023-10-11T10:54:23.386 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3000 IOs, 1296788480 bytes 2023-10-11T10:54:23.472 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3100 IOs, 1323995136 bytes 2023-10-11T10:54:23.638 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3200 IOs, 1360830464 bytes 2023-10-11T10:54:23.756 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3300 IOs, 1380549632 bytes 2023-10-11T10:54:23.908 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3400 IOs, 1408724992 bytes 2023-10-11T10:54:24.219 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3500 IOs, 1534375936 bytes 2023-10-11T10:54:24.401 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3600 IOs, 1578426368 bytes 2023-10-11T10:54:24.616 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3700 IOs, 1644270592 bytes 2023-10-11T10:54:24.803 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3800 IOs, 1720046592 bytes 2023-10-11T10:54:24.924 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3900 IOs, 1764779008 bytes 2023-10-11T10:54:25.226 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4000 IOs, 1842124800 bytes 2023-10-11T10:54:25.322 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4100 IOs, 1854995456 bytes 2023-10-11T10:54:25.476 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4200 IOs, 1892843520 bytes 2023-10-11T10:54:25.698 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4300 IOs, 1964131328 bytes 2023-10-11T10:54:25.800 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4400 IOs, 1965851648 bytes 2023-10-11T10:54:25.911 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4500 IOs, 1981639680 bytes 2023-10-11T10:54:26.020 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4600 IOs, 1986056192 bytes 2023-10-11T10:54:26.098 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4700 IOs, 1986671616 bytes 2023-10-11T10:54:26.226 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4800 IOs, 2006903808 bytes 2023-10-11T10:54:26.352 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4900 IOs, 2033901568 bytes 2023-10-11T10:54:26.601 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5000 IOs, 2119897088 bytes 2023-10-11T10:54:26.787 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5100 IOs, 2174184448 bytes 2023-10-11T10:54:27.021 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5200 IOs, 2244017152 bytes 2023-10-11T10:54:27.178 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5300 IOs, 2280327168 bytes 2023-10-11T10:54:27.321 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5400 IOs, 2311139328 bytes 2023-10-11T10:54:27.496 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5500 IOs, 2362332160 bytes 2023-10-11T10:54:27.648 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5600 IOs, 2392588288 bytes 2023-10-11T10:54:27.793 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5700 IOs, 2423094272 bytes 2023-10-11T10:54:27.902 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5800 IOs, 2436469760 bytes 2023-10-11T10:54:28.070 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5900 IOs, 2476756992 bytes 2023-10-11T10:54:28.218 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6000 IOs, 2508088320 bytes 2023-10-11T10:54:28.313 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6100 IOs, 2516234240 bytes 2023-10-11T10:54:28.451 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6200 IOs, 2548413440 bytes 2023-10-11T10:54:28.781 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6300 IOs, 2676603904 bytes 2023-10-11T10:54:28.919 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6400 IOs, 2734896128 bytes 2023-10-11T10:54:29.087 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6500 IOs, 2767946752 bytes 2023-10-11T10:54:29.215 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6600 IOs, 2783724544 bytes 2023-10-11T10:54:29.322 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6700 IOs, 2797919232 bytes 2023-10-11T10:54:29.457 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6800 IOs, 2824154112 bytes 2023-10-11T10:54:29.565 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6900 IOs, 2839670784 bytes 2023-10-11T10:54:29.654 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:7000 IOs, 2841950208 bytes 2023-10-11T10:54:29.782 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:7100 IOs, 2865008640 bytes 2023-10-11T10:54:29.980 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:7200 IOs, 2926467072 bytes 2023-10-11T10:54:30.118 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:7300 IOs, 2948263936 bytes 2023-10-11T10:54:30.397 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ error_code=124 2023-10-11T10:54:30.397 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ set -e 2023-10-11T10:54:30.397 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' 124 -eq 124 ']' 2023-10-11T10:54:30.397 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T10:54:30.398 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T10:54:30.398 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_3 2023-10-11T10:54:30.398 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ create_image_and_enable_mirror cluster2 mirror image_3 snapshot 128M 2023-10-11T10:54:30.398 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:54:30.399 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:54:30.399 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:54:30.399 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:54:30.399 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_3 2023-10-11T10:54:30.399 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:54:30.399 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local mode=snapshot 2023-10-11T10:54:30.400 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n snapshot ']' 2023-10-11T10:54:30.400 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:54:30.400 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ create_image cluster2 mirror image_3 128M 2023-10-11T10:54:30.400 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:54:30.400 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:54:30.400 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:54:30.401 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:54:30.401 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_3 2023-10-11T10:54:30.401 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:54:30.401 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local size=128 2023-10-11T10:54:30.401 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n 128M ']' 2023-10-11T10:54:30.402 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ size=128M 2023-10-11T10:54:30.402 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:54:30.402 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_3 2023-10-11T10:54:30.444 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' image = image ']' 2023-10-11T10:54:30.444 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ enable_mirror cluster2 mirror image_3 snapshot 2023-10-11T10:54:30.444 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:54:30.444 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:54:30.444 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_3 2023-10-11T10:54:30.445 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local mode=snapshot 2023-10-11T10:54:30.445 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_3 snapshot 2023-10-11T10:54:30.994 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:Mirroring enabled 2023-10-11T10:54:31.002 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 info mirror/image_3 2023-10-11T10:54:31.040 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:rbd image 'image_3': 2023-10-11T10:54:31.040 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: size 128 MiB in 32 objects 2023-10-11T10:54:31.040 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: order 22 (4 MiB objects) 2023-10-11T10:54:31.040 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: snapshot_count: 1 2023-10-11T10:54:31.041 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: id: 122c3df1a420 2023-10-11T10:54:31.041 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: block_name_prefix: rbd_data.122c3df1a420 2023-10-11T10:54:31.041 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: format: 2 2023-10-11T10:54:31.041 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-10-11T10:54:31.042 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: op_features: 2023-10-11T10:54:31.042 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: flags: 2023-10-11T10:54:31.042 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: create_timestamp: Wed Oct 11 10:54:30 2023 2023-10-11T10:54:31.043 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: access_timestamp: Wed Oct 11 10:54:30 2023 2023-10-11T10:54:31.043 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: modify_timestamp: Wed Oct 11 10:54:30 2023 2023-10-11T10:54:31.043 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring state: enabled 2023-10-11T10:54:31.043 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring mode: snapshot 2023-10-11T10:54:31.044 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring global id: 665a8ec8-563e-4515-ba87-41db0aac57ac 2023-10-11T10:54:31.044 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring primary: true 2023-10-11T10:54:31.045 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n '' ']' 2023-10-11T10:54:31.045 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ stress_write_image cluster2 mirror image_3 2023-10-11T10:54:31.046 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:54:31.046 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:54:31.046 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_3 2023-10-11T10:54:31.047 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-10-11T10:54:31.047 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local duration=5 2023-10-11T10:54:31.047 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ set +e 2023-10-11T10:54:31.047 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_3 --debug-rbd=20 --debug-journaler=20 2023-10-11T10:54:31.231 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:100 IOs, 30540800 bytes 2023-10-11T10:54:31.340 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:200 IOs, 45032448 bytes 2023-10-11T10:54:31.523 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:300 IOs, 95048704 bytes 2023-10-11T10:54:31.745 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:400 IOs, 165771264 bytes 2023-10-11T10:54:32.006 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:500 IOs, 257530880 bytes 2023-10-11T10:54:32.254 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:600 IOs, 342508544 bytes 2023-10-11T10:54:32.429 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:700 IOs, 394133504 bytes 2023-10-11T10:54:32.563 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:800 IOs, 421540864 bytes 2023-10-11T10:54:32.828 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:900 IOs, 522544128 bytes 2023-10-11T10:54:32.939 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1000 IOs, 537567232 bytes 2023-10-11T10:54:33.140 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1100 IOs, 599312384 bytes 2023-10-11T10:54:33.261 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1200 IOs, 619799552 bytes 2023-10-11T10:54:33.408 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1300 IOs, 655659008 bytes 2023-10-11T10:54:33.609 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1400 IOs, 715734016 bytes 2023-10-11T10:54:33.706 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1500 IOs, 724400128 bytes 2023-10-11T10:54:33.871 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1600 IOs, 739732480 bytes 2023-10-11T10:54:34.445 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1700 IOs, 818375680 bytes 2023-10-11T10:54:34.573 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1800 IOs, 843842560 bytes 2023-10-11T10:54:34.768 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1900 IOs, 867153920 bytes 2023-10-11T10:54:34.920 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2000 IOs, 914529280 bytes 2023-10-11T10:54:35.171 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2100 IOs, 982874112 bytes 2023-10-11T10:54:35.361 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2200 IOs, 1036324864 bytes 2023-10-11T10:54:35.492 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2300 IOs, 1074202624 bytes 2023-10-11T10:54:35.668 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2400 IOs, 1130679296 bytes 2023-10-11T10:54:35.888 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2500 IOs, 1187279872 bytes 2023-10-11T10:54:36.025 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2600 IOs, 1211858944 bytes 2023-10-11T10:54:36.050 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ error_code=124 2023-10-11T10:54:36.050 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ set -e 2023-10-11T10:54:36.050 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' 124 -eq 124 ']' 2023-10-11T10:54:36.050 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T10:54:36.050 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T10:54:36.051 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_4 2023-10-11T10:54:36.051 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ create_image_and_enable_mirror cluster2 mirror image_4 snapshot 128M 2023-10-11T10:54:36.051 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:54:36.051 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:54:36.051 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:54:36.052 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:54:36.052 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_4 2023-10-11T10:54:36.052 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:54:36.052 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local mode=snapshot 2023-10-11T10:54:36.052 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n snapshot ']' 2023-10-11T10:54:36.053 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:54:36.053 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ create_image cluster2 mirror image_4 128M 2023-10-11T10:54:36.053 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:54:36.053 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:54:36.053 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:54:36.054 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:54:36.054 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_4 2023-10-11T10:54:36.054 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:54:36.054 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local size=128 2023-10-11T10:54:36.054 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n 128M ']' 2023-10-11T10:54:36.055 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ size=128M 2023-10-11T10:54:36.055 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:54:36.055 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_4 2023-10-11T10:54:36.095 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' image = image ']' 2023-10-11T10:54:36.095 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ enable_mirror cluster2 mirror image_4 snapshot 2023-10-11T10:54:36.096 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:54:36.096 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:54:36.096 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_4 2023-10-11T10:54:36.096 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local mode=snapshot 2023-10-11T10:54:36.097 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_4 snapshot 2023-10-11T10:54:36.318 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:Mirroring enabled 2023-10-11T10:54:36.329 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 info mirror/image_4 2023-10-11T10:54:36.364 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:rbd image 'image_4': 2023-10-11T10:54:36.364 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: size 128 MiB in 32 objects 2023-10-11T10:54:36.364 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: order 22 (4 MiB objects) 2023-10-11T10:54:36.365 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: snapshot_count: 1 2023-10-11T10:54:36.365 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: id: 1234af770e98 2023-10-11T10:54:36.365 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: block_name_prefix: rbd_data.1234af770e98 2023-10-11T10:54:36.365 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: format: 2 2023-10-11T10:54:36.365 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-10-11T10:54:36.366 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: op_features: 2023-10-11T10:54:36.366 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: flags: 2023-10-11T10:54:36.366 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: create_timestamp: Wed Oct 11 10:54:36 2023 2023-10-11T10:54:36.366 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: access_timestamp: Wed Oct 11 10:54:36 2023 2023-10-11T10:54:36.367 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: modify_timestamp: Wed Oct 11 10:54:36 2023 2023-10-11T10:54:36.367 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring state: enabled 2023-10-11T10:54:36.367 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring mode: snapshot 2023-10-11T10:54:36.367 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring global id: f031088c-5560-4c60-b77f-f68d8adaedd0 2023-10-11T10:54:36.367 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring primary: true 2023-10-11T10:54:36.368 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n '' ']' 2023-10-11T10:54:36.368 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ stress_write_image cluster2 mirror image_4 2023-10-11T10:54:36.369 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:54:36.369 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:54:36.369 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_4 2023-10-11T10:54:36.369 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-10-11T10:54:36.370 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local duration=5 2023-10-11T10:54:36.370 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ set +e 2023-10-11T10:54:36.370 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_4 --debug-rbd=20 --debug-journaler=20 2023-10-11T10:54:36.701 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:100 IOs, 101960704 bytes 2023-10-11T10:54:36.825 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:200 IOs, 114593792 bytes 2023-10-11T10:54:36.937 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:300 IOs, 135424000 bytes 2023-10-11T10:54:37.068 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:400 IOs, 160703488 bytes 2023-10-11T10:54:37.462 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:500 IOs, 298458112 bytes 2023-10-11T10:54:37.613 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:600 IOs, 345797632 bytes 2023-10-11T10:54:37.942 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:700 IOs, 449203200 bytes 2023-10-11T10:54:38.039 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:800 IOs, 454306816 bytes 2023-10-11T10:54:38.192 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:900 IOs, 495009792 bytes 2023-10-11T10:54:38.373 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1000 IOs, 539332608 bytes 2023-10-11T10:54:38.507 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1100 IOs, 566231040 bytes 2023-10-11T10:54:38.680 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1200 IOs, 608243712 bytes 2023-10-11T10:54:38.907 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1300 IOs, 689446912 bytes 2023-10-11T10:54:39.013 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1400 IOs, 702334976 bytes 2023-10-11T10:54:39.164 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1500 IOs, 733843456 bytes 2023-10-11T10:54:39.430 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1600 IOs, 815435776 bytes 2023-10-11T10:54:39.696 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1700 IOs, 913952768 bytes 2023-10-11T10:54:39.841 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1800 IOs, 939282432 bytes 2023-10-11T10:54:39.991 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1900 IOs, 973731840 bytes 2023-10-11T10:54:40.207 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2000 IOs, 1046124544 bytes 2023-10-11T10:54:40.399 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2100 IOs, 1106714624 bytes 2023-10-11T10:54:40.573 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2200 IOs, 1157104640 bytes 2023-10-11T10:54:40.732 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2300 IOs, 1195896832 bytes 2023-10-11T10:54:40.827 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2400 IOs, 1202758656 bytes 2023-10-11T10:54:40.947 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2500 IOs, 1219427328 bytes 2023-10-11T10:54:41.061 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2600 IOs, 1236449280 bytes 2023-10-11T10:54:41.191 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2700 IOs, 1264913408 bytes 2023-10-11T10:54:41.292 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2800 IOs, 1273766912 bytes 2023-10-11T10:54:41.378 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ error_code=124 2023-10-11T10:54:41.378 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ set -e 2023-10-11T10:54:41.379 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' 124 -eq 124 ']' 2023-10-11T10:54:41.379 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T10:54:41.379 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T10:54:41.379 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_5 2023-10-11T10:54:41.380 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ create_image_and_enable_mirror cluster2 mirror image_5 snapshot 128M 2023-10-11T10:54:41.380 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:54:41.380 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:54:41.380 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:54:41.380 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:54:41.381 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_5 2023-10-11T10:54:41.381 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:54:41.381 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local mode=snapshot 2023-10-11T10:54:41.381 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n snapshot ']' 2023-10-11T10:54:41.382 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:54:41.382 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ create_image cluster2 mirror image_5 128M 2023-10-11T10:54:41.382 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:54:41.383 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:54:41.383 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:54:41.383 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:54:41.383 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_5 2023-10-11T10:54:41.383 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:54:41.384 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local size=128 2023-10-11T10:54:41.384 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n 128M ']' 2023-10-11T10:54:41.384 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ size=128M 2023-10-11T10:54:41.384 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:54:41.385 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_5 2023-10-11T10:54:41.425 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' image = image ']' 2023-10-11T10:54:41.426 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ enable_mirror cluster2 mirror image_5 snapshot 2023-10-11T10:54:41.426 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:54:41.426 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:54:41.427 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_5 2023-10-11T10:54:41.427 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local mode=snapshot 2023-10-11T10:54:41.427 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_5 snapshot 2023-10-11T10:54:42.012 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:Mirroring enabled 2023-10-11T10:54:42.020 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 info mirror/image_5 2023-10-11T10:54:42.056 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:rbd image 'image_5': 2023-10-11T10:54:42.057 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: size 128 MiB in 32 objects 2023-10-11T10:54:42.057 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: order 22 (4 MiB objects) 2023-10-11T10:54:42.057 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: snapshot_count: 1 2023-10-11T10:54:42.057 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: id: 123cb0dc7388 2023-10-11T10:54:42.058 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: block_name_prefix: rbd_data.123cb0dc7388 2023-10-11T10:54:42.058 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: format: 2 2023-10-11T10:54:42.058 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-10-11T10:54:42.058 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: op_features: 2023-10-11T10:54:42.058 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: flags: 2023-10-11T10:54:42.059 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: create_timestamp: Wed Oct 11 10:54:41 2023 2023-10-11T10:54:42.059 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: access_timestamp: Wed Oct 11 10:54:41 2023 2023-10-11T10:54:42.059 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: modify_timestamp: Wed Oct 11 10:54:41 2023 2023-10-11T10:54:42.059 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring state: enabled 2023-10-11T10:54:42.060 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring mode: snapshot 2023-10-11T10:54:42.060 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring global id: a9ff47ee-c486-4ecb-981d-63d0dafd14e5 2023-10-11T10:54:42.060 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring primary: true 2023-10-11T10:54:42.061 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n '' ']' 2023-10-11T10:54:42.062 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ stress_write_image cluster2 mirror image_5 2023-10-11T10:54:42.062 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:54:42.062 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:54:42.062 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_5 2023-10-11T10:54:42.063 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-10-11T10:54:42.063 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local duration=9 2023-10-11T10:54:42.063 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ set +e 2023-10-11T10:54:42.063 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_5 --debug-rbd=20 --debug-journaler=20 2023-10-11T10:54:42.302 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:100 IOs, 46479360 bytes 2023-10-11T10:54:42.400 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:200 IOs, 52596736 bytes 2023-10-11T10:54:42.589 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:300 IOs, 112052224 bytes 2023-10-11T10:54:42.750 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:400 IOs, 168920064 bytes 2023-10-11T10:54:42.841 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:500 IOs, 176568320 bytes 2023-10-11T10:54:43.024 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:600 IOs, 221743104 bytes 2023-10-11T10:54:43.207 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:700 IOs, 269185024 bytes 2023-10-11T10:54:43.348 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:800 IOs, 299714560 bytes 2023-10-11T10:54:43.554 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:900 IOs, 384551936 bytes 2023-10-11T10:54:43.738 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1000 IOs, 421730304 bytes 2023-10-11T10:54:43.863 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1100 IOs, 447725568 bytes 2023-10-11T10:54:43.960 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1200 IOs, 455682048 bytes 2023-10-11T10:54:44.090 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1300 IOs, 473466880 bytes 2023-10-11T10:54:44.218 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1400 IOs, 517812224 bytes 2023-10-11T10:54:44.399 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1500 IOs, 541637632 bytes 2023-10-11T10:54:44.546 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1600 IOs, 572881920 bytes 2023-10-11T10:54:44.734 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1700 IOs, 625889280 bytes 2023-10-11T10:54:44.916 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1800 IOs, 695461888 bytes 2023-10-11T10:54:45.103 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1900 IOs, 742265856 bytes 2023-10-11T10:54:45.334 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2000 IOs, 836167680 bytes 2023-10-11T10:54:45.502 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2100 IOs, 860293120 bytes 2023-10-11T10:54:45.751 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2200 IOs, 947134464 bytes 2023-10-11T10:54:45.906 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2300 IOs, 975819776 bytes 2023-10-11T10:54:46.091 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2400 IOs, 1022098432 bytes 2023-10-11T10:54:46.321 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2500 IOs, 1097211904 bytes 2023-10-11T10:54:46.431 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2600 IOs, 1112110080 bytes 2023-10-11T10:54:46.598 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2700 IOs, 1150628864 bytes 2023-10-11T10:54:46.787 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2800 IOs, 1201867776 bytes 2023-10-11T10:54:46.932 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2900 IOs, 1227311104 bytes 2023-10-11T10:54:47.084 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3000 IOs, 1259504640 bytes 2023-10-11T10:54:47.252 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3100 IOs, 1311782912 bytes 2023-10-11T10:54:47.402 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3200 IOs, 1344702464 bytes 2023-10-11T10:54:47.501 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3300 IOs, 1346934784 bytes 2023-10-11T10:54:47.618 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3400 IOs, 1391022080 bytes 2023-10-11T10:54:47.865 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3500 IOs, 1450473472 bytes 2023-10-11T10:54:48.106 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3600 IOs, 1531721728 bytes 2023-10-11T10:54:48.286 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3700 IOs, 1582012416 bytes 2023-10-11T10:54:48.462 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3800 IOs, 1627796480 bytes 2023-10-11T10:54:48.545 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3900 IOs, 1631675392 bytes 2023-10-11T10:54:48.665 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4000 IOs, 1651769344 bytes 2023-10-11T10:54:48.810 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4100 IOs, 1687007232 bytes 2023-10-11T10:54:49.021 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4200 IOs, 1755077632 bytes 2023-10-11T10:54:49.107 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4300 IOs, 1762041856 bytes 2023-10-11T10:54:49.236 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4400 IOs, 1789581312 bytes 2023-10-11T10:54:49.374 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4500 IOs, 1803727872 bytes 2023-10-11T10:54:49.539 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4600 IOs, 1847115776 bytes 2023-10-11T10:54:49.712 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4700 IOs, 1912900608 bytes 2023-10-11T10:54:49.952 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4800 IOs, 1990931456 bytes 2023-10-11T10:54:50.214 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4900 IOs, 2072475648 bytes 2023-10-11T10:54:50.373 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5000 IOs, 2107034624 bytes 2023-10-11T10:54:50.466 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5100 IOs, 2113751040 bytes 2023-10-11T10:54:50.610 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5200 IOs, 2142750720 bytes 2023-10-11T10:54:50.759 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5300 IOs, 2176847872 bytes 2023-10-11T10:54:50.916 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5400 IOs, 2215500800 bytes 2023-10-11T10:54:51.066 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ error_code=124 2023-10-11T10:54:51.066 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ set -e 2023-10-11T10:54:51.066 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' 124 -eq 124 ']' 2023-10-11T10:54:51.067 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T10:54:51.067 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T10:54:51.067 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_6 2023-10-11T10:54:51.067 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ create_image_and_enable_mirror cluster2 mirror image_6 snapshot 128M 2023-10-11T10:54:51.067 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:54:51.068 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:54:51.068 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:54:51.068 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:54:51.068 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_6 2023-10-11T10:54:51.068 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:54:51.069 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local mode=snapshot 2023-10-11T10:54:51.069 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n snapshot ']' 2023-10-11T10:54:51.069 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:54:51.069 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ create_image cluster2 mirror image_6 128M 2023-10-11T10:54:51.069 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:54:51.069 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:54:51.070 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:54:51.070 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:54:51.070 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_6 2023-10-11T10:54:51.070 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:54:51.070 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local size=128 2023-10-11T10:54:51.071 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n 128M ']' 2023-10-11T10:54:51.071 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ size=128M 2023-10-11T10:54:51.071 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:54:51.071 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_6 2023-10-11T10:54:51.112 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' image = image ']' 2023-10-11T10:54:51.113 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ enable_mirror cluster2 mirror image_6 snapshot 2023-10-11T10:54:51.113 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:54:51.113 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:54:51.113 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_6 2023-10-11T10:54:51.113 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local mode=snapshot 2023-10-11T10:54:51.114 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_6 snapshot 2023-10-11T10:54:52.013 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:Mirroring enabled 2023-10-11T10:54:52.023 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 info mirror/image_6 2023-10-11T10:54:52.059 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:rbd image 'image_6': 2023-10-11T10:54:52.059 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: size 128 MiB in 32 objects 2023-10-11T10:54:52.059 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: order 22 (4 MiB objects) 2023-10-11T10:54:52.060 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: snapshot_count: 1 2023-10-11T10:54:52.060 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: id: 1244b518755 2023-10-11T10:54:52.060 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: block_name_prefix: rbd_data.1244b518755 2023-10-11T10:54:52.060 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: format: 2 2023-10-11T10:54:52.060 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-10-11T10:54:52.061 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: op_features: 2023-10-11T10:54:52.061 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: flags: 2023-10-11T10:54:52.061 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: create_timestamp: Wed Oct 11 10:54:51 2023 2023-10-11T10:54:52.061 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: access_timestamp: Wed Oct 11 10:54:51 2023 2023-10-11T10:54:52.061 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: modify_timestamp: Wed Oct 11 10:54:51 2023 2023-10-11T10:54:52.062 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring state: enabled 2023-10-11T10:54:52.062 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring mode: snapshot 2023-10-11T10:54:52.062 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring global id: 7ce830b7-0256-4d8e-905d-503bebab5873 2023-10-11T10:54:52.062 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring primary: true 2023-10-11T10:54:52.063 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n '' ']' 2023-10-11T10:54:52.063 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ stress_write_image cluster2 mirror image_6 2023-10-11T10:54:52.063 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:54:52.064 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:54:52.064 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_6 2023-10-11T10:54:52.064 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-10-11T10:54:52.065 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local duration=9 2023-10-11T10:54:52.065 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ set +e 2023-10-11T10:54:52.065 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_6 --debug-rbd=20 --debug-journaler=20 2023-10-11T10:54:52.298 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:100 IOs, 32140288 bytes 2023-10-11T10:54:52.477 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:200 IOs, 102177792 bytes 2023-10-11T10:54:52.658 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:300 IOs, 149039104 bytes 2023-10-11T10:54:52.839 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:400 IOs, 187778048 bytes 2023-10-11T10:54:53.063 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:500 IOs, 247753728 bytes 2023-10-11T10:54:53.212 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:600 IOs, 279113728 bytes 2023-10-11T10:54:53.342 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:700 IOs, 308222976 bytes 2023-10-11T10:54:53.592 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:800 IOs, 368897024 bytes 2023-10-11T10:54:53.723 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:900 IOs, 394432512 bytes 2023-10-11T10:54:53.931 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1000 IOs, 453498880 bytes 2023-10-11T10:54:54.116 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1100 IOs, 510404608 bytes 2023-10-11T10:54:54.292 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1200 IOs, 563131392 bytes 2023-10-11T10:54:54.456 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1300 IOs, 598502400 bytes 2023-10-11T10:54:54.577 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1400 IOs, 654042112 bytes 2023-10-11T10:54:54.828 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1500 IOs, 692806656 bytes 2023-10-11T10:54:55.014 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1600 IOs, 754268160 bytes 2023-10-11T10:54:55.278 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1700 IOs, 824930304 bytes 2023-10-11T10:54:55.524 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1800 IOs, 898639872 bytes 2023-10-11T10:54:55.712 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1900 IOs, 953077760 bytes 2023-10-11T10:54:55.864 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2000 IOs, 993161216 bytes 2023-10-11T10:54:56.171 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2100 IOs, 1053792256 bytes 2023-10-11T10:54:56.348 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2200 IOs, 1121131520 bytes 2023-10-11T10:54:56.564 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2300 IOs, 1170385920 bytes 2023-10-11T10:54:56.716 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2400 IOs, 1206781952 bytes 2023-10-11T10:54:56.928 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2500 IOs, 1279003648 bytes 2023-10-11T10:54:57.137 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2600 IOs, 1334608896 bytes 2023-10-11T10:54:57.361 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2700 IOs, 1386558464 bytes 2023-10-11T10:54:57.518 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2800 IOs, 1424375808 bytes 2023-10-11T10:54:57.767 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2900 IOs, 1515846656 bytes 2023-10-11T10:54:57.925 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3000 IOs, 1551385600 bytes 2023-10-11T10:54:58.166 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3100 IOs, 1637370880 bytes 2023-10-11T10:54:58.358 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3200 IOs, 1686327296 bytes 2023-10-11T10:54:58.450 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3300 IOs, 1691383808 bytes 2023-10-11T10:54:58.628 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3400 IOs, 1728493568 bytes 2023-10-11T10:54:58.761 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3500 IOs, 1754856448 bytes 2023-10-11T10:54:58.890 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3600 IOs, 1789490176 bytes 2023-10-11T10:54:59.050 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3700 IOs, 1819680768 bytes 2023-10-11T10:54:59.164 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3800 IOs, 1837814784 bytes 2023-10-11T10:54:59.348 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3900 IOs, 1884847104 bytes 2023-10-11T10:54:59.589 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4000 IOs, 2007459840 bytes 2023-10-11T10:54:59.807 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4100 IOs, 2031044608 bytes 2023-10-11T10:54:59.966 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4200 IOs, 2066167808 bytes 2023-10-11T10:55:00.135 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4300 IOs, 2110955520 bytes 2023-10-11T10:55:00.266 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4400 IOs, 2134763520 bytes 2023-10-11T10:55:00.384 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4500 IOs, 2146721792 bytes 2023-10-11T10:55:00.479 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4600 IOs, 2174644224 bytes 2023-10-11T10:55:00.649 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4700 IOs, 2196700160 bytes 2023-10-11T10:55:00.854 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4800 IOs, 2257005568 bytes 2023-10-11T10:55:00.977 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4900 IOs, 2292362240 bytes 2023-10-11T10:55:01.070 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ error_code=124 2023-10-11T10:55:01.070 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ set -e 2023-10-11T10:55:01.070 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' 124 -eq 124 ']' 2023-10-11T10:55:01.070 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T10:55:01.070 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T10:55:01.071 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_7 2023-10-11T10:55:01.071 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ create_image_and_enable_mirror cluster2 mirror image_7 snapshot 128M 2023-10-11T10:55:01.071 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:55:01.071 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:55:01.071 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:55:01.072 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:55:01.072 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_7 2023-10-11T10:55:01.072 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:55:01.072 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local mode=snapshot 2023-10-11T10:55:01.072 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n snapshot ']' 2023-10-11T10:55:01.072 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:55:01.073 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ create_image cluster2 mirror image_7 128M 2023-10-11T10:55:01.073 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:55:01.073 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:55:01.073 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:55:01.073 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:55:01.074 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_7 2023-10-11T10:55:01.074 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:55:01.074 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local size=128 2023-10-11T10:55:01.074 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n 128M ']' 2023-10-11T10:55:01.074 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ size=128M 2023-10-11T10:55:01.075 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:55:01.075 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_7 2023-10-11T10:55:01.114 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' image = image ']' 2023-10-11T10:55:01.115 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ enable_mirror cluster2 mirror image_7 snapshot 2023-10-11T10:55:01.115 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:55:01.115 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:55:01.115 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_7 2023-10-11T10:55:01.115 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local mode=snapshot 2023-10-11T10:55:01.115 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_7 snapshot 2023-10-11T10:55:02.046 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:Mirroring enabled 2023-10-11T10:55:02.055 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 info mirror/image_7 2023-10-11T10:55:02.094 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:rbd image 'image_7': 2023-10-11T10:55:02.094 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: size 128 MiB in 32 objects 2023-10-11T10:55:02.095 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: order 22 (4 MiB objects) 2023-10-11T10:55:02.095 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: snapshot_count: 1 2023-10-11T10:55:02.095 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: id: 124d94033fa8 2023-10-11T10:55:02.095 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: block_name_prefix: rbd_data.124d94033fa8 2023-10-11T10:55:02.095 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: format: 2 2023-10-11T10:55:02.096 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-10-11T10:55:02.096 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: op_features: 2023-10-11T10:55:02.096 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: flags: 2023-10-11T10:55:02.096 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: create_timestamp: Wed Oct 11 10:55:01 2023 2023-10-11T10:55:02.096 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: access_timestamp: Wed Oct 11 10:55:01 2023 2023-10-11T10:55:02.096 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: modify_timestamp: Wed Oct 11 10:55:01 2023 2023-10-11T10:55:02.097 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring state: enabled 2023-10-11T10:55:02.097 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring mode: snapshot 2023-10-11T10:55:02.097 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring global id: 96782a85-6dfc-4ac8-9a0a-1c21292c349e 2023-10-11T10:55:02.097 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring primary: true 2023-10-11T10:55:02.098 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n '' ']' 2023-10-11T10:55:02.098 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ stress_write_image cluster2 mirror image_7 2023-10-11T10:55:02.099 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:55:02.099 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:55:02.099 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_7 2023-10-11T10:55:02.099 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-10-11T10:55:02.100 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local duration=12 2023-10-11T10:55:02.100 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ set +e 2023-10-11T10:55:02.100 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_7 --debug-rbd=20 --debug-journaler=20 2023-10-11T10:55:02.265 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:100 IOs, 5351424 bytes 2023-10-11T10:55:02.416 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:200 IOs, 56124416 bytes 2023-10-11T10:55:02.554 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:300 IOs, 72742912 bytes 2023-10-11T10:55:02.762 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:400 IOs, 128829440 bytes 2023-10-11T10:55:02.946 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:500 IOs, 188206080 bytes 2023-10-11T10:55:03.106 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:600 IOs, 230899712 bytes 2023-10-11T10:55:03.342 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:700 IOs, 314957824 bytes 2023-10-11T10:55:03.517 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:800 IOs, 350601216 bytes 2023-10-11T10:55:03.667 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:900 IOs, 383315968 bytes 2023-10-11T10:55:03.808 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1000 IOs, 434546688 bytes 2023-10-11T10:55:04.009 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1100 IOs, 463209472 bytes 2023-10-11T10:55:04.194 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1200 IOs, 511791104 bytes 2023-10-11T10:55:04.335 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1300 IOs, 542015488 bytes 2023-10-11T10:55:04.584 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1400 IOs, 599931904 bytes 2023-10-11T10:55:04.863 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1500 IOs, 695213056 bytes 2023-10-11T10:55:05.246 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1600 IOs, 810468352 bytes 2023-10-11T10:55:05.407 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1700 IOs, 855344128 bytes 2023-10-11T10:55:05.567 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1800 IOs, 904479744 bytes 2023-10-11T10:55:05.751 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1900 IOs, 959361024 bytes 2023-10-11T10:55:05.864 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2000 IOs, 975634432 bytes 2023-10-11T10:55:05.977 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2100 IOs, 996579328 bytes 2023-10-11T10:55:06.105 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2200 IOs, 1021891584 bytes 2023-10-11T10:55:06.254 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2300 IOs, 1056257024 bytes 2023-10-11T10:55:06.435 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2400 IOs, 1111081984 bytes 2023-10-11T10:55:06.522 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2500 IOs, 1111639040 bytes 2023-10-11T10:55:06.637 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2600 IOs, 1128471552 bytes 2023-10-11T10:55:06.893 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2700 IOs, 1216178176 bytes 2023-10-11T10:55:07.091 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2800 IOs, 1272067072 bytes 2023-10-11T10:55:07.241 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2900 IOs, 1301091328 bytes 2023-10-11T10:55:07.389 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3000 IOs, 1332684800 bytes 2023-10-11T10:55:07.481 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3100 IOs, 1346961408 bytes 2023-10-11T10:55:07.578 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3200 IOs, 1347556352 bytes 2023-10-11T10:55:07.810 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3300 IOs, 1429626880 bytes 2023-10-11T10:55:08.026 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3400 IOs, 1501407232 bytes 2023-10-11T10:55:08.177 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3500 IOs, 1535757312 bytes 2023-10-11T10:55:08.323 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3600 IOs, 1573306368 bytes 2023-10-11T10:55:08.594 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3700 IOs, 1680067584 bytes 2023-10-11T10:55:08.802 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3800 IOs, 1750180864 bytes 2023-10-11T10:55:08.984 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3900 IOs, 1772034048 bytes 2023-10-11T10:55:09.157 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4000 IOs, 1820539904 bytes 2023-10-11T10:55:09.375 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4100 IOs, 1892714496 bytes 2023-10-11T10:55:09.633 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4200 IOs, 1972622336 bytes 2023-10-11T10:55:09.730 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4300 IOs, 1973250048 bytes 2023-10-11T10:55:09.932 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4400 IOs, 2029755392 bytes 2023-10-11T10:55:10.133 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4500 IOs, 2106700800 bytes 2023-10-11T10:55:10.348 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4600 IOs, 2156097536 bytes 2023-10-11T10:55:10.615 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4700 IOs, 2249382912 bytes 2023-10-11T10:55:10.727 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4800 IOs, 2262511616 bytes 2023-10-11T10:55:10.872 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4900 IOs, 2291408896 bytes 2023-10-11T10:55:11.030 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5000 IOs, 2331891712 bytes 2023-10-11T10:55:11.186 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5100 IOs, 2364716032 bytes 2023-10-11T10:55:11.322 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5200 IOs, 2398469120 bytes 2023-10-11T10:55:11.467 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5300 IOs, 2422421504 bytes 2023-10-11T10:55:11.610 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5400 IOs, 2461663232 bytes 2023-10-11T10:55:11.724 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5500 IOs, 2462266368 bytes 2023-10-11T10:55:11.918 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5600 IOs, 2510700544 bytes 2023-10-11T10:55:12.006 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5700 IOs, 2511363072 bytes 2023-10-11T10:55:12.326 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5800 IOs, 2538468352 bytes 2023-10-11T10:55:12.562 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5900 IOs, 2613798912 bytes 2023-10-11T10:55:12.692 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6000 IOs, 2634994688 bytes 2023-10-11T10:55:12.876 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6100 IOs, 2680899584 bytes 2023-10-11T10:55:13.081 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6200 IOs, 2728709120 bytes 2023-10-11T10:55:13.248 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6300 IOs, 2772861952 bytes 2023-10-11T10:55:13.436 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6400 IOs, 2818939904 bytes 2023-10-11T10:55:13.574 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6500 IOs, 2864368640 bytes 2023-10-11T10:55:13.827 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6600 IOs, 2926555136 bytes 2023-10-11T10:55:13.940 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6700 IOs, 2963535872 bytes 2023-10-11T10:55:14.105 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ error_code=124 2023-10-11T10:55:14.106 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ set -e 2023-10-11T10:55:14.106 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' 124 -eq 124 ']' 2023-10-11T10:55:14.106 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T10:55:14.106 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T10:55:14.107 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_8 2023-10-11T10:55:14.107 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ create_image_and_enable_mirror cluster2 mirror image_8 snapshot 128M 2023-10-11T10:55:14.107 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:55:14.107 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:55:14.107 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:55:14.108 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:55:14.108 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_8 2023-10-11T10:55:14.108 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:55:14.108 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local mode=snapshot 2023-10-11T10:55:14.108 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n snapshot ']' 2023-10-11T10:55:14.108 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:55:14.109 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ create_image cluster2 mirror image_8 128M 2023-10-11T10:55:14.109 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:55:14.109 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:55:14.109 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:55:14.109 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:55:14.109 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_8 2023-10-11T10:55:14.110 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:55:14.110 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local size=128 2023-10-11T10:55:14.110 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n 128M ']' 2023-10-11T10:55:14.110 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ size=128M 2023-10-11T10:55:14.110 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:55:14.111 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_8 2023-10-11T10:55:14.149 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' image = image ']' 2023-10-11T10:55:14.149 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ enable_mirror cluster2 mirror image_8 snapshot 2023-10-11T10:55:14.150 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:55:14.150 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:55:14.150 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_8 2023-10-11T10:55:14.150 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local mode=snapshot 2023-10-11T10:55:14.150 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_8 snapshot 2023-10-11T10:55:14.324 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:Mirroring enabled 2023-10-11T10:55:14.331 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 info mirror/image_8 2023-10-11T10:55:14.367 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:rbd image 'image_8': 2023-10-11T10:55:14.367 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: size 128 MiB in 32 objects 2023-10-11T10:55:14.367 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: order 22 (4 MiB objects) 2023-10-11T10:55:14.368 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: snapshot_count: 1 2023-10-11T10:55:14.368 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: id: 1256209def44 2023-10-11T10:55:14.368 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: block_name_prefix: rbd_data.1256209def44 2023-10-11T10:55:14.368 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: format: 2 2023-10-11T10:55:14.368 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-10-11T10:55:14.369 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: op_features: 2023-10-11T10:55:14.369 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: flags: 2023-10-11T10:55:14.369 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: create_timestamp: Wed Oct 11 10:55:14 2023 2023-10-11T10:55:14.369 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: access_timestamp: Wed Oct 11 10:55:14 2023 2023-10-11T10:55:14.369 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: modify_timestamp: Wed Oct 11 10:55:14 2023 2023-10-11T10:55:14.370 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring state: enabled 2023-10-11T10:55:14.370 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring mode: snapshot 2023-10-11T10:55:14.370 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring global id: ff8c48f9-2b0b-4783-bc85-bfb3318837fb 2023-10-11T10:55:14.370 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring primary: true 2023-10-11T10:55:14.371 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n '' ']' 2023-10-11T10:55:14.372 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ stress_write_image cluster2 mirror image_8 2023-10-11T10:55:14.372 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:55:14.372 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:55:14.373 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_8 2023-10-11T10:55:14.373 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-10-11T10:55:14.373 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local duration=11 2023-10-11T10:55:14.373 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ set +e 2023-10-11T10:55:14.373 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_8 --debug-rbd=20 --debug-journaler=20 2023-10-11T10:55:14.529 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.3844850899886151) > weight (0.25) 2023-10-11T10:55:14.529 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.1 2023-10-11T10:55:14.530 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Sent signal 15 2023-10-11T10:55:14.530 INFO:tasks.rbd_mirror_thrash:waiting for 4 secs before reviving daemons 2023-10-11T10:55:14.531 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi111.stderr:2023-10-11T10:55:14.530+0000 7f2432e08700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 106583) UID: 1000 2023-10-11T10:55:14.591 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:100 IOs, 44518400 bytes 2023-10-11T10:55:14.770 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:200 IOs, 90833920 bytes 2023-10-11T10:55:14.972 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:300 IOs, 170170368 bytes 2023-10-11T10:55:15.132 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:400 IOs, 190982144 bytes 2023-10-11T10:55:15.314 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:500 IOs, 242995200 bytes 2023-10-11T10:55:15.474 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:600 IOs, 300418048 bytes 2023-10-11T10:55:15.585 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:700 IOs, 302789632 bytes 2023-10-11T10:55:15.828 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:800 IOs, 397322240 bytes 2023-10-11T10:55:16.019 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:900 IOs, 429385728 bytes 2023-10-11T10:55:16.175 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1000 IOs, 460871680 bytes 2023-10-11T10:55:16.488 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1100 IOs, 471729152 bytes 2023-10-11T10:55:16.616 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1200 IOs, 485617664 bytes 2023-10-11T10:55:16.722 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1300 IOs, 498550784 bytes 2023-10-11T10:55:16.888 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1400 IOs, 537647104 bytes 2023-10-11T10:55:17.017 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1500 IOs, 574404608 bytes 2023-10-11T10:55:17.207 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1600 IOs, 632425472 bytes 2023-10-11T10:55:17.528 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1700 IOs, 645951488 bytes 2023-10-11T10:55:17.660 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1800 IOs, 669423616 bytes 2023-10-11T10:55:17.800 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1900 IOs, 693668864 bytes 2023-10-11T10:55:17.977 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2000 IOs, 738998272 bytes 2023-10-11T10:55:18.223 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2100 IOs, 818759680 bytes 2023-10-11T10:55:18.462 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2200 IOs, 887384064 bytes 2023-10-11T10:55:18.531 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.1 2023-10-11T10:55:18.532 INFO:teuthology.orchestra.run:waiting for 600 2023-10-11T10:55:18.532 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.1 2023-10-11T10:55:18.532 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2023-10-11T10:55:18.533 DEBUG:teuthology.orchestra.run.smithi111:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2023-10-11T10:55:18.536 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2023-10-11T10:55:18.536 INFO:tasks.rbd_mirror_thrash:waiting for 96 secs before thrashing 2023-10-11T10:55:18.619 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2300 IOs, 917729280 bytes 2023-10-11T10:55:18.832 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2400 IOs, 982289408 bytes 2023-10-11T10:55:18.987 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2500 IOs, 1012196352 bytes 2023-10-11T10:55:19.158 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2600 IOs, 1052846080 bytes 2023-10-11T10:55:19.350 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2700 IOs, 1111109632 bytes 2023-10-11T10:55:19.572 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2800 IOs, 1179445248 bytes 2023-10-11T10:55:19.658 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2900 IOs, 1180052480 bytes 2023-10-11T10:55:19.668 INFO:tasks.ceph.cluster2.osd.1.smithi111.stderr:2023-10-11T10:55:19.667+0000 7f9628714700 -1 reset not still connected to 0x564246a251e0 2023-10-11T10:55:19.672 INFO:tasks.ceph.cluster2.osd.1.smithi111.stderr:2023-10-11T10:55:19.669+0000 7f9628714700 -1 reset not still connected to 0x564251b57ad0 2023-10-11T10:55:19.672 INFO:tasks.ceph.cluster2.osd.1.smithi111.stderr:2023-10-11T10:55:19.669+0000 7f9628714700 -1 reset not still connected to 0x564256ba31e0 2023-10-11T10:55:19.845 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3000 IOs, 1238103040 bytes 2023-10-11T10:55:20.001 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3100 IOs, 1260447744 bytes 2023-10-11T10:55:20.232 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3200 IOs, 1349902336 bytes 2023-10-11T10:55:20.420 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3300 IOs, 1395608576 bytes 2023-10-11T10:55:20.579 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3400 IOs, 1444587520 bytes 2023-10-11T10:55:20.699 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3500 IOs, 1458354176 bytes 2023-10-11T10:55:20.827 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3600 IOs, 1482169344 bytes 2023-10-11T10:55:20.980 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3700 IOs, 1516123136 bytes 2023-10-11T10:55:21.125 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3800 IOs, 1577264128 bytes 2023-10-11T10:55:21.377 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3900 IOs, 1639282688 bytes 2023-10-11T10:55:21.609 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4000 IOs, 1712507904 bytes 2023-10-11T10:55:21.790 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4100 IOs, 1752627200 bytes 2023-10-11T10:55:21.893 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4200 IOs, 1759046656 bytes 2023-10-11T10:55:22.116 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4300 IOs, 1824202752 bytes 2023-10-11T10:55:22.277 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4400 IOs, 1860736000 bytes 2023-10-11T10:55:22.442 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4500 IOs, 1913753600 bytes 2023-10-11T10:55:22.623 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4600 IOs, 1950683136 bytes 2023-10-11T10:55:22.795 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4700 IOs, 1991044096 bytes 2023-10-11T10:55:22.927 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4800 IOs, 2014816256 bytes 2023-10-11T10:55:23.081 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4900 IOs, 2057336832 bytes 2023-10-11T10:55:23.235 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5000 IOs, 2090482688 bytes 2023-10-11T10:55:23.413 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5100 IOs, 2156112896 bytes 2023-10-11T10:55:23.587 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5200 IOs, 2178384896 bytes 2023-10-11T10:55:23.725 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5300 IOs, 2202808320 bytes 2023-10-11T10:55:23.905 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5400 IOs, 2241587200 bytes 2023-10-11T10:55:24.101 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5500 IOs, 2288274432 bytes 2023-10-11T10:55:24.215 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5600 IOs, 2304764928 bytes 2023-10-11T10:55:24.353 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5700 IOs, 2330729472 bytes 2023-10-11T10:55:24.484 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5800 IOs, 2350820352 bytes 2023-10-11T10:55:24.634 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5900 IOs, 2384540672 bytes 2023-10-11T10:55:24.749 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6000 IOs, 2398175232 bytes 2023-10-11T10:55:24.853 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6100 IOs, 2403359744 bytes 2023-10-11T10:55:25.049 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6200 IOs, 2448892928 bytes 2023-10-11T10:55:25.241 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6300 IOs, 2493285376 bytes 2023-10-11T10:55:25.379 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ error_code=124 2023-10-11T10:55:25.379 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ set -e 2023-10-11T10:55:25.379 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' 124 -eq 124 ']' 2023-10-11T10:55:25.380 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T10:55:25.380 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T10:55:25.380 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_9 2023-10-11T10:55:25.380 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ create_image_and_enable_mirror cluster2 mirror image_9 snapshot 128M 2023-10-11T10:55:25.380 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:55:25.381 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:55:25.381 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:55:25.381 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:55:25.381 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_9 2023-10-11T10:55:25.381 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:55:25.381 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local mode=snapshot 2023-10-11T10:55:25.382 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n snapshot ']' 2023-10-11T10:55:25.382 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:55:25.382 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ create_image cluster2 mirror image_9 128M 2023-10-11T10:55:25.382 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:55:25.382 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:55:25.383 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:55:25.383 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:55:25.383 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_9 2023-10-11T10:55:25.383 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:55:25.383 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local size=128 2023-10-11T10:55:25.384 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n 128M ']' 2023-10-11T10:55:25.384 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ size=128M 2023-10-11T10:55:25.384 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:55:25.384 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_9 2023-10-11T10:55:25.429 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' image = image ']' 2023-10-11T10:55:25.429 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ enable_mirror cluster2 mirror image_9 snapshot 2023-10-11T10:55:25.429 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:55:25.429 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:55:25.430 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_9 2023-10-11T10:55:25.430 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local mode=snapshot 2023-10-11T10:55:25.430 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_9 snapshot 2023-10-11T10:55:26.411 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:Mirroring enabled 2023-10-11T10:55:26.424 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 info mirror/image_9 2023-10-11T10:55:26.462 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:rbd image 'image_9': 2023-10-11T10:55:26.463 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: size 128 MiB in 32 objects 2023-10-11T10:55:26.463 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: order 22 (4 MiB objects) 2023-10-11T10:55:26.463 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: snapshot_count: 1 2023-10-11T10:55:26.463 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: id: 1263396c185e 2023-10-11T10:55:26.464 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: block_name_prefix: rbd_data.1263396c185e 2023-10-11T10:55:26.464 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: format: 2 2023-10-11T10:55:26.464 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-10-11T10:55:26.465 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: op_features: 2023-10-11T10:55:26.465 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: flags: 2023-10-11T10:55:26.465 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: create_timestamp: Wed Oct 11 10:55:25 2023 2023-10-11T10:55:26.465 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: access_timestamp: Wed Oct 11 10:55:25 2023 2023-10-11T10:55:26.466 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: modify_timestamp: Wed Oct 11 10:55:25 2023 2023-10-11T10:55:26.466 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring state: enabled 2023-10-11T10:55:26.466 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring mode: snapshot 2023-10-11T10:55:26.466 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring global id: ca5ef4e0-7cfe-484d-8d61-93b7e745a919 2023-10-11T10:55:26.467 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring primary: true 2023-10-11T10:55:26.468 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n '' ']' 2023-10-11T10:55:26.468 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ stress_write_image cluster2 mirror image_9 2023-10-11T10:55:26.468 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:55:26.468 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:55:26.468 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_9 2023-10-11T10:55:26.469 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-10-11T10:55:26.469 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local duration=5 2023-10-11T10:55:26.469 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ set +e 2023-10-11T10:55:26.469 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_9 --debug-rbd=20 --debug-journaler=20 2023-10-11T10:55:26.709 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:100 IOs, 43255808 bytes 2023-10-11T10:55:26.911 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:200 IOs, 119422976 bytes 2023-10-11T10:55:27.117 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:300 IOs, 167252992 bytes 2023-10-11T10:55:27.368 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:400 IOs, 268547072 bytes 2023-10-11T10:55:27.557 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:500 IOs, 302065664 bytes 2023-10-11T10:55:27.817 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:600 IOs, 394362880 bytes 2023-10-11T10:55:27.974 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:700 IOs, 438259712 bytes 2023-10-11T10:55:28.210 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:800 IOs, 505986048 bytes 2023-10-11T10:55:28.366 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:900 IOs, 541219840 bytes 2023-10-11T10:55:28.588 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1000 IOs, 615463936 bytes 2023-10-11T10:55:28.810 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1100 IOs, 687771648 bytes 2023-10-11T10:55:28.911 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1200 IOs, 694235136 bytes 2023-10-11T10:55:29.080 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1300 IOs, 745525248 bytes 2023-10-11T10:55:29.252 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1400 IOs, 784688128 bytes 2023-10-11T10:55:29.545 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1500 IOs, 901281792 bytes 2023-10-11T10:55:29.769 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1600 IOs, 979619840 bytes 2023-10-11T10:55:30.080 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1700 IOs, 1102130176 bytes 2023-10-11T10:55:30.231 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1800 IOs, 1130101760 bytes 2023-10-11T10:55:30.462 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1900 IOs, 1194314752 bytes 2023-10-11T10:55:30.654 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2000 IOs, 1269543936 bytes 2023-10-11T10:55:30.835 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2100 IOs, 1298344960 bytes 2023-10-11T10:55:31.002 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2200 IOs, 1332143104 bytes 2023-10-11T10:55:31.133 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2300 IOs, 1360474112 bytes 2023-10-11T10:55:31.319 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2400 IOs, 1415058432 bytes 2023-10-11T10:55:31.416 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2500 IOs, 1425975296 bytes 2023-10-11T10:55:31.479 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ error_code=124 2023-10-11T10:55:31.479 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ set -e 2023-10-11T10:55:31.479 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' 124 -eq 124 ']' 2023-10-11T10:55:31.480 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T10:55:31.480 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T10:55:31.480 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_10 2023-10-11T10:55:31.480 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ create_image_and_enable_mirror cluster2 mirror image_10 snapshot 128M 2023-10-11T10:55:31.481 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:55:31.481 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:55:31.481 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:55:31.481 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:55:31.481 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_10 2023-10-11T10:55:31.481 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:55:31.482 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local mode=snapshot 2023-10-11T10:55:31.482 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n snapshot ']' 2023-10-11T10:55:31.482 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:55:31.482 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ create_image cluster2 mirror image_10 128M 2023-10-11T10:55:31.482 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:55:31.483 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:55:31.483 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:55:31.483 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:55:31.483 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_10 2023-10-11T10:55:31.483 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:55:31.483 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local size=128 2023-10-11T10:55:31.484 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n 128M ']' 2023-10-11T10:55:31.484 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ size=128M 2023-10-11T10:55:31.484 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:55:31.484 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_10 2023-10-11T10:55:31.526 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' image = image ']' 2023-10-11T10:55:31.526 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ enable_mirror cluster2 mirror image_10 snapshot 2023-10-11T10:55:31.526 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:55:31.526 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:55:31.527 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_10 2023-10-11T10:55:31.527 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local mode=snapshot 2023-10-11T10:55:31.527 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_10 snapshot 2023-10-11T10:55:32.062 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:Mirroring enabled 2023-10-11T10:55:32.071 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 info mirror/image_10 2023-10-11T10:55:32.108 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:rbd image 'image_10': 2023-10-11T10:55:32.108 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: size 128 MiB in 32 objects 2023-10-11T10:55:32.109 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: order 22 (4 MiB objects) 2023-10-11T10:55:32.109 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: snapshot_count: 1 2023-10-11T10:55:32.109 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: id: 126b395f083 2023-10-11T10:55:32.109 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: block_name_prefix: rbd_data.126b395f083 2023-10-11T10:55:32.109 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: format: 2 2023-10-11T10:55:32.109 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-10-11T10:55:32.110 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: op_features: 2023-10-11T10:55:32.110 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: flags: 2023-10-11T10:55:32.110 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: create_timestamp: Wed Oct 11 10:55:31 2023 2023-10-11T10:55:32.110 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: access_timestamp: Wed Oct 11 10:55:31 2023 2023-10-11T10:55:32.110 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: modify_timestamp: Wed Oct 11 10:55:31 2023 2023-10-11T10:55:32.111 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring state: enabled 2023-10-11T10:55:32.111 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring mode: snapshot 2023-10-11T10:55:32.111 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring global id: 965cf2d0-179c-4721-be88-950f8a10b7ae 2023-10-11T10:55:32.111 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring primary: true 2023-10-11T10:55:32.112 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n '' ']' 2023-10-11T10:55:32.112 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ stress_write_image cluster2 mirror image_10 2023-10-11T10:55:32.112 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:55:32.113 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:55:32.113 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_10 2023-10-11T10:55:32.113 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-10-11T10:55:32.113 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local duration=11 2023-10-11T10:55:32.114 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ set +e 2023-10-11T10:55:32.114 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_10 --debug-rbd=20 --debug-journaler=20 2023-10-11T10:55:32.565 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:100 IOs, 45511680 bytes 2023-10-11T10:55:32.675 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:200 IOs, 62792704 bytes 2023-10-11T10:55:32.816 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:300 IOs, 86299648 bytes 2023-10-11T10:55:32.907 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:400 IOs, 86903808 bytes 2023-10-11T10:55:33.099 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:500 IOs, 138945536 bytes 2023-10-11T10:55:33.231 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:600 IOs, 162732032 bytes 2023-10-11T10:55:33.501 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:700 IOs, 258715648 bytes 2023-10-11T10:55:33.592 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:800 IOs, 259338240 bytes 2023-10-11T10:55:33.736 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:900 IOs, 295092224 bytes 2023-10-11T10:55:33.964 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1000 IOs, 343906304 bytes 2023-10-11T10:55:34.162 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1100 IOs, 397462528 bytes 2023-10-11T10:55:34.313 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1200 IOs, 429657088 bytes 2023-10-11T10:55:34.554 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1300 IOs, 503927808 bytes 2023-10-11T10:55:34.810 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1400 IOs, 611258368 bytes 2023-10-11T10:55:35.068 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1500 IOs, 681276416 bytes 2023-10-11T10:55:35.156 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1600 IOs, 681875456 bytes 2023-10-11T10:55:35.311 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1700 IOs, 714936320 bytes 2023-10-11T10:55:35.467 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1800 IOs, 754624512 bytes 2023-10-11T10:55:35.639 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1900 IOs, 813402112 bytes 2023-10-11T10:55:35.767 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2000 IOs, 828409856 bytes 2023-10-11T10:55:36.009 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2100 IOs, 888870912 bytes 2023-10-11T10:55:36.222 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2200 IOs, 953708544 bytes 2023-10-11T10:55:36.353 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2300 IOs, 971819008 bytes 2023-10-11T10:55:36.517 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2400 IOs, 1014623232 bytes 2023-10-11T10:55:36.654 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2500 IOs, 1030331392 bytes 2023-10-11T10:55:36.867 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2600 IOs, 1092548608 bytes 2023-10-11T10:55:37.064 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2700 IOs, 1147117568 bytes 2023-10-11T10:55:37.315 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2800 IOs, 1234698240 bytes 2023-10-11T10:55:37.560 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2900 IOs, 1313805312 bytes 2023-10-11T10:55:37.729 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3000 IOs, 1353262080 bytes 2023-10-11T10:55:37.933 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3100 IOs, 1403967488 bytes 2023-10-11T10:55:38.066 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3200 IOs, 1418816512 bytes 2023-10-11T10:55:38.256 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3300 IOs, 1474868224 bytes 2023-10-11T10:55:38.498 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3400 IOs, 1559432192 bytes 2023-10-11T10:55:38.532 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.7947925895285817) > weight (0.25) 2023-10-11T10:55:38.533 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.6907266749121242) > weight (0.25) 2023-10-11T10:55:38.533 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.30380903468496223) > weight (0.25) 2023-10-11T10:55:38.533 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.3 2023-10-11T10:55:38.534 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Sent signal 15 2023-10-11T10:55:38.534 INFO:tasks.rbd_mirror_thrash:waiting for 5 secs before reviving daemons 2023-10-11T10:55:38.534 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi111.stderr:2023-10-11T10:55:38.534+0000 7f2782301700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 (PID: 106696) UID: 1000 2023-10-11T10:55:38.684 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3500 IOs, 1613168640 bytes 2023-10-11T10:55:38.838 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3600 IOs, 1633064960 bytes 2023-10-11T10:55:38.939 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3700 IOs, 1641594880 bytes 2023-10-11T10:55:39.109 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3800 IOs, 1683768320 bytes 2023-10-11T10:55:39.305 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3900 IOs, 1739112448 bytes 2023-10-11T10:55:39.515 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4000 IOs, 1798754304 bytes 2023-10-11T10:55:39.628 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4100 IOs, 1815726080 bytes 2023-10-11T10:55:39.723 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4200 IOs, 1830760448 bytes 2023-10-11T10:55:39.939 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4300 IOs, 1880880128 bytes 2023-10-11T10:55:40.029 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4400 IOs, 1881501696 bytes 2023-10-11T10:55:40.165 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4500 IOs, 1928976384 bytes 2023-10-11T10:55:40.536 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4600 IOs, 2047092736 bytes 2023-10-11T10:55:40.671 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4700 IOs, 2078215168 bytes 2023-10-11T10:55:40.812 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4800 IOs, 2100675584 bytes 2023-10-11T10:55:41.031 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4900 IOs, 2167207936 bytes 2023-10-11T10:55:41.194 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5000 IOs, 2200952832 bytes 2023-10-11T10:55:41.310 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5100 IOs, 2224022528 bytes 2023-10-11T10:55:41.436 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5200 IOs, 2240257024 bytes 2023-10-11T10:55:41.642 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5300 IOs, 2306236416 bytes 2023-10-11T10:55:41.833 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5400 IOs, 2366898176 bytes 2023-10-11T10:55:42.031 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5500 IOs, 2412900352 bytes 2023-10-11T10:55:42.289 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5600 IOs, 2486789120 bytes 2023-10-11T10:55:42.426 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5700 IOs, 2504685568 bytes 2023-10-11T10:55:42.568 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5800 IOs, 2529705984 bytes 2023-10-11T10:55:42.724 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5900 IOs, 2559423488 bytes 2023-10-11T10:55:42.905 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6000 IOs, 2613941248 bytes 2023-10-11T10:55:43.062 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6100 IOs, 2639278080 bytes 2023-10-11T10:55:43.127 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ error_code=124 2023-10-11T10:55:43.127 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ set -e 2023-10-11T10:55:43.128 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' 124 -eq 124 ']' 2023-10-11T10:55:43.128 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T10:55:43.128 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T10:55:43.128 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_11 2023-10-11T10:55:43.128 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ create_image_and_enable_mirror cluster2 mirror image_11 snapshot 128M 2023-10-11T10:55:43.129 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:55:43.129 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:55:43.129 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:55:43.129 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:55:43.129 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_11 2023-10-11T10:55:43.130 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:55:43.130 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local mode=snapshot 2023-10-11T10:55:43.130 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n snapshot ']' 2023-10-11T10:55:43.130 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:55:43.130 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ create_image cluster2 mirror image_11 128M 2023-10-11T10:55:43.131 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:55:43.131 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:55:43.131 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:55:43.131 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:55:43.131 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_11 2023-10-11T10:55:43.132 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:55:43.132 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local size=128 2023-10-11T10:55:43.132 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n 128M ']' 2023-10-11T10:55:43.132 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ size=128M 2023-10-11T10:55:43.132 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:55:43.132 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_11 2023-10-11T10:55:43.176 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' image = image ']' 2023-10-11T10:55:43.177 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ enable_mirror cluster2 mirror image_11 snapshot 2023-10-11T10:55:43.177 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:55:43.177 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:55:43.177 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_11 2023-10-11T10:55:43.177 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local mode=snapshot 2023-10-11T10:55:43.177 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_11 snapshot 2023-10-11T10:55:43.336 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:Mirroring enabled 2023-10-11T10:55:43.346 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 info mirror/image_11 2023-10-11T10:55:43.381 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:rbd image 'image_11': 2023-10-11T10:55:43.381 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: size 128 MiB in 32 objects 2023-10-11T10:55:43.382 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: order 22 (4 MiB objects) 2023-10-11T10:55:43.382 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: snapshot_count: 1 2023-10-11T10:55:43.382 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: id: 1274e47b4d9d 2023-10-11T10:55:43.382 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: block_name_prefix: rbd_data.1274e47b4d9d 2023-10-11T10:55:43.382 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: format: 2 2023-10-11T10:55:43.383 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-10-11T10:55:43.383 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: op_features: 2023-10-11T10:55:43.383 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: flags: 2023-10-11T10:55:43.383 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: create_timestamp: Wed Oct 11 10:55:43 2023 2023-10-11T10:55:43.383 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: access_timestamp: Wed Oct 11 10:55:43 2023 2023-10-11T10:55:43.384 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: modify_timestamp: Wed Oct 11 10:55:43 2023 2023-10-11T10:55:43.384 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring state: enabled 2023-10-11T10:55:43.384 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring mode: snapshot 2023-10-11T10:55:43.384 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring global id: 35668704-fbeb-4c06-998e-79dc6d9e7e60 2023-10-11T10:55:43.384 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring primary: true 2023-10-11T10:55:43.385 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n '' ']' 2023-10-11T10:55:43.385 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ stress_write_image cluster2 mirror image_11 2023-10-11T10:55:43.386 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:55:43.386 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:55:43.386 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_11 2023-10-11T10:55:43.386 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-10-11T10:55:43.387 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local duration=9 2023-10-11T10:55:43.387 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ set +e 2023-10-11T10:55:43.387 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_11 --debug-rbd=20 --debug-journaler=20 2023-10-11T10:55:43.535 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.3 2023-10-11T10:55:43.535 INFO:teuthology.orchestra.run:waiting for 600 2023-10-11T10:55:43.535 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.3 2023-10-11T10:55:43.535 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Restarting daemon 2023-10-11T10:55:43.536 DEBUG:teuthology.orchestra.run.smithi111:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 2023-10-11T10:55:43.539 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Started 2023-10-11T10:55:43.539 INFO:tasks.rbd_mirror_thrash:waiting for 95 secs before thrashing 2023-10-11T10:55:43.651 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:100 IOs, 59917312 bytes 2023-10-11T10:55:43.786 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:200 IOs, 78332928 bytes 2023-10-11T10:55:43.959 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:300 IOs, 109379584 bytes 2023-10-11T10:55:44.114 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:400 IOs, 133905408 bytes 2023-10-11T10:55:44.219 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:500 IOs, 145602560 bytes 2023-10-11T10:55:44.360 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:600 IOs, 172054528 bytes 2023-10-11T10:55:44.494 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:700 IOs, 209527808 bytes 2023-10-11T10:55:44.800 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:800 IOs, 305042432 bytes 2023-10-11T10:55:44.887 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:900 IOs, 305798144 bytes 2023-10-11T10:55:44.981 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1000 IOs, 306395136 bytes 2023-10-11T10:55:45.164 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1100 IOs, 374636544 bytes 2023-10-11T10:55:45.331 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1200 IOs, 390552576 bytes 2023-10-11T10:55:45.415 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1300 IOs, 393209856 bytes 2023-10-11T10:55:45.552 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1400 IOs, 420253696 bytes 2023-10-11T10:55:45.674 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1500 IOs, 426532864 bytes 2023-10-11T10:55:45.857 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1600 IOs, 479584256 bytes 2023-10-11T10:55:46.025 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1700 IOs, 518658048 bytes 2023-10-11T10:55:46.183 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1800 IOs, 563427328 bytes 2023-10-11T10:55:46.312 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1900 IOs, 582897664 bytes 2023-10-11T10:55:46.416 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2000 IOs, 586813440 bytes 2023-10-11T10:55:46.517 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2100 IOs, 597256192 bytes 2023-10-11T10:55:46.660 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2200 IOs, 630679552 bytes 2023-10-11T10:55:46.793 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2300 IOs, 652908544 bytes 2023-10-11T10:55:47.030 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2400 IOs, 736519168 bytes 2023-10-11T10:55:47.249 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2500 IOs, 795507712 bytes 2023-10-11T10:55:47.337 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2600 IOs, 796122112 bytes 2023-10-11T10:55:47.492 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2700 IOs, 838792192 bytes 2023-10-11T10:55:47.617 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2800 IOs, 854205440 bytes 2023-10-11T10:55:47.785 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2900 IOs, 898755584 bytes 2023-10-11T10:55:47.922 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3000 IOs, 924933120 bytes 2023-10-11T10:55:48.079 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3100 IOs, 970194944 bytes 2023-10-11T10:55:48.334 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3200 IOs, 1054763008 bytes 2023-10-11T10:55:48.438 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3300 IOs, 1071218688 bytes 2023-10-11T10:55:48.535 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3400 IOs, 1097100288 bytes 2023-10-11T10:55:48.743 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3500 IOs, 1131819008 bytes 2023-10-11T10:55:48.927 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3600 IOs, 1195088896 bytes 2023-10-11T10:55:49.165 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3700 IOs, 1254807552 bytes 2023-10-11T10:55:49.433 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3800 IOs, 1358496768 bytes 2023-10-11T10:55:49.674 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3900 IOs, 1420250112 bytes 2023-10-11T10:55:49.917 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4000 IOs, 1509156864 bytes 2023-10-11T10:55:50.085 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4100 IOs, 1528206336 bytes 2023-10-11T10:55:50.196 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4200 IOs, 1560835072 bytes 2023-10-11T10:55:50.349 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4300 IOs, 1579758592 bytes 2023-10-11T10:55:50.464 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4400 IOs, 1596023808 bytes 2023-10-11T10:55:50.787 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4500 IOs, 1721692160 bytes 2023-10-11T10:55:50.894 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4600 IOs, 1722318848 bytes 2023-10-11T10:55:51.096 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4700 IOs, 1782248448 bytes 2023-10-11T10:55:51.273 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4800 IOs, 1823984640 bytes 2023-10-11T10:55:51.513 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4900 IOs, 1894651904 bytes 2023-10-11T10:55:51.621 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5000 IOs, 1911020544 bytes 2023-10-11T10:55:51.917 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5100 IOs, 2019262464 bytes 2023-10-11T10:55:52.135 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5200 IOs, 2072613888 bytes 2023-10-11T10:55:52.216 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5300 IOs, 2073220096 bytes 2023-10-11T10:55:52.394 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ error_code=124 2023-10-11T10:55:52.394 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ set -e 2023-10-11T10:55:52.395 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' 124 -eq 124 ']' 2023-10-11T10:55:52.395 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T10:55:52.395 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T10:55:52.395 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_12 2023-10-11T10:55:52.395 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ create_image_and_enable_mirror cluster2 mirror image_12 snapshot 128M 2023-10-11T10:55:52.396 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:55:52.396 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:55:52.396 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:55:52.396 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:55:52.396 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_12 2023-10-11T10:55:52.397 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:55:52.397 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local mode=snapshot 2023-10-11T10:55:52.397 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n snapshot ']' 2023-10-11T10:55:52.397 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:55:52.397 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ create_image cluster2 mirror image_12 128M 2023-10-11T10:55:52.398 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:55:52.398 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:55:52.398 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:55:52.398 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:55:52.398 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_12 2023-10-11T10:55:52.398 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:55:52.399 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local size=128 2023-10-11T10:55:52.399 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n 128M ']' 2023-10-11T10:55:52.399 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ size=128M 2023-10-11T10:55:52.399 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:55:52.399 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_12 2023-10-11T10:55:52.443 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' image = image ']' 2023-10-11T10:55:52.443 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ enable_mirror cluster2 mirror image_12 snapshot 2023-10-11T10:55:52.444 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:55:52.444 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:55:52.444 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_12 2023-10-11T10:55:52.444 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local mode=snapshot 2023-10-11T10:55:52.444 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_12 snapshot 2023-10-11T10:55:53.397 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:Mirroring enabled 2023-10-11T10:55:53.407 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 info mirror/image_12 2023-10-11T10:55:53.444 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:rbd image 'image_12': 2023-10-11T10:55:53.445 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: size 128 MiB in 32 objects 2023-10-11T10:55:53.445 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: order 22 (4 MiB objects) 2023-10-11T10:55:53.445 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: snapshot_count: 1 2023-10-11T10:55:53.445 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: id: 12899cae3f26 2023-10-11T10:55:53.446 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: block_name_prefix: rbd_data.12899cae3f26 2023-10-11T10:55:53.446 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: format: 2 2023-10-11T10:55:53.446 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-10-11T10:55:53.446 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: op_features: 2023-10-11T10:55:53.446 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: flags: 2023-10-11T10:55:53.447 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: create_timestamp: Wed Oct 11 10:55:52 2023 2023-10-11T10:55:53.447 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: access_timestamp: Wed Oct 11 10:55:52 2023 2023-10-11T10:55:53.447 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: modify_timestamp: Wed Oct 11 10:55:52 2023 2023-10-11T10:55:53.447 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring state: enabled 2023-10-11T10:55:53.447 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring mode: snapshot 2023-10-11T10:55:53.448 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring global id: 418ab5b5-2cf6-4b23-a645-1106d804c432 2023-10-11T10:55:53.448 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring primary: true 2023-10-11T10:55:53.449 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n '' ']' 2023-10-11T10:55:53.449 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ stress_write_image cluster2 mirror image_12 2023-10-11T10:55:53.449 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:55:53.449 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:55:53.449 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_12 2023-10-11T10:55:53.450 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-10-11T10:55:53.450 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local duration=12 2023-10-11T10:55:53.450 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ set +e 2023-10-11T10:55:53.451 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_12 --debug-rbd=20 --debug-journaler=20 2023-10-11T10:55:53.689 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:100 IOs, 60729344 bytes 2023-10-11T10:55:53.920 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:200 IOs, 126515200 bytes 2023-10-11T10:55:54.039 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:300 IOs, 137928704 bytes 2023-10-11T10:55:54.164 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:400 IOs, 155586560 bytes 2023-10-11T10:55:54.318 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:500 IOs, 180579328 bytes 2023-10-11T10:55:54.474 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:600 IOs, 211395584 bytes 2023-10-11T10:55:54.627 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:700 IOs, 236227584 bytes 2023-10-11T10:55:54.725 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:800 IOs, 236857344 bytes 2023-10-11T10:55:54.888 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:900 IOs, 283015168 bytes 2023-10-11T10:55:55.055 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1000 IOs, 328671232 bytes 2023-10-11T10:55:55.211 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1100 IOs, 363475968 bytes 2023-10-11T10:55:55.344 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1200 IOs, 410156032 bytes 2023-10-11T10:55:55.498 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1300 IOs, 453989376 bytes 2023-10-11T10:55:55.692 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1400 IOs, 480268288 bytes 2023-10-11T10:55:55.842 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1500 IOs, 515088384 bytes 2023-10-11T10:55:55.963 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1600 IOs, 532835328 bytes 2023-10-11T10:55:56.127 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1700 IOs, 575934464 bytes 2023-10-11T10:55:56.227 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1800 IOs, 584842240 bytes 2023-10-11T10:55:56.467 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1900 IOs, 674573312 bytes 2023-10-11T10:55:56.648 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2000 IOs, 723181568 bytes 2023-10-11T10:55:56.804 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2100 IOs, 747727872 bytes 2023-10-11T10:55:57.017 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2200 IOs, 807895040 bytes 2023-10-11T10:55:57.160 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2300 IOs, 833117184 bytes 2023-10-11T10:55:57.299 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2400 IOs, 854378496 bytes 2023-10-11T10:55:57.485 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2500 IOs, 905756672 bytes 2023-10-11T10:55:57.631 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2600 IOs, 929991680 bytes 2023-10-11T10:55:57.799 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2700 IOs, 969332736 bytes 2023-10-11T10:55:57.899 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2800 IOs, 983619584 bytes 2023-10-11T10:55:58.139 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2900 IOs, 1062311936 bytes 2023-10-11T10:55:58.330 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3000 IOs, 1114186752 bytes 2023-10-11T10:55:58.510 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3100 IOs, 1177900032 bytes 2023-10-11T10:55:58.756 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3200 IOs, 1251816448 bytes 2023-10-11T10:55:58.981 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3300 IOs, 1319802880 bytes 2023-10-11T10:55:59.203 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3400 IOs, 1368335360 bytes 2023-10-11T10:55:59.358 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3500 IOs, 1404870656 bytes 2023-10-11T10:55:59.542 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3600 IOs, 1453019136 bytes 2023-10-11T10:55:59.664 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3700 IOs, 1470473216 bytes 2023-10-11T10:55:59.899 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3800 IOs, 1541071872 bytes 2023-10-11T10:56:00.114 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3900 IOs, 1611467776 bytes 2023-10-11T10:56:00.321 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4000 IOs, 1680136192 bytes 2023-10-11T10:56:00.539 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4100 IOs, 1750658048 bytes 2023-10-11T10:56:00.750 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4200 IOs, 1810627584 bytes 2023-10-11T10:56:00.961 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4300 IOs, 1873161216 bytes 2023-10-11T10:56:01.154 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4400 IOs, 1940275200 bytes 2023-10-11T10:56:01.352 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4500 IOs, 1982236672 bytes 2023-10-11T10:56:01.508 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4600 IOs, 2019907584 bytes 2023-10-11T10:56:01.617 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4700 IOs, 2034162688 bytes 2023-10-11T10:56:01.763 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4800 IOs, 2064093184 bytes 2023-10-11T10:56:01.862 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4900 IOs, 2064684032 bytes 2023-10-11T10:56:02.087 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5000 IOs, 2132234240 bytes 2023-10-11T10:56:02.299 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5100 IOs, 2189416448 bytes 2023-10-11T10:56:02.477 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5200 IOs, 2228149248 bytes 2023-10-11T10:56:02.713 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5300 IOs, 2330299392 bytes 2023-10-11T10:56:02.981 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5400 IOs, 2399168512 bytes 2023-10-11T10:56:03.172 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5500 IOs, 2465821696 bytes 2023-10-11T10:56:03.327 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5600 IOs, 2484346880 bytes 2023-10-11T10:56:03.573 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5700 IOs, 2582516736 bytes 2023-10-11T10:56:03.780 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5800 IOs, 2621967360 bytes 2023-10-11T10:56:03.889 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5900 IOs, 2629249024 bytes 2023-10-11T10:56:04.121 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6000 IOs, 2701745152 bytes 2023-10-11T10:56:04.307 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6100 IOs, 2755368960 bytes 2023-10-11T10:56:04.424 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6200 IOs, 2783184896 bytes 2023-10-11T10:56:04.600 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6300 IOs, 2819539968 bytes 2023-10-11T10:56:04.809 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6400 IOs, 2880619520 bytes 2023-10-11T10:56:05.061 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6500 IOs, 2968137728 bytes 2023-10-11T10:56:05.201 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6600 IOs, 2993215488 bytes 2023-10-11T10:56:05.413 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6700 IOs, 3058111488 bytes 2023-10-11T10:56:05.456 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ error_code=124 2023-10-11T10:56:05.456 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ set -e 2023-10-11T10:56:05.456 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' 124 -eq 124 ']' 2023-10-11T10:56:05.456 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T10:56:05.457 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T10:56:05.457 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_13 2023-10-11T10:56:05.457 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ create_image_and_enable_mirror cluster2 mirror image_13 snapshot 128M 2023-10-11T10:56:05.457 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:56:05.457 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:56:05.457 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:56:05.458 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:56:05.458 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_13 2023-10-11T10:56:05.458 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:56:05.458 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local mode=snapshot 2023-10-11T10:56:05.458 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n snapshot ']' 2023-10-11T10:56:05.459 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:56:05.459 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ create_image cluster2 mirror image_13 128M 2023-10-11T10:56:05.459 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:56:05.459 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:56:05.459 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:56:05.459 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:56:05.460 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_13 2023-10-11T10:56:05.460 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:56:05.460 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local size=128 2023-10-11T10:56:05.460 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n 128M ']' 2023-10-11T10:56:05.460 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ size=128M 2023-10-11T10:56:05.460 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:56:05.461 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_13 2023-10-11T10:56:05.500 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' image = image ']' 2023-10-11T10:56:05.501 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ enable_mirror cluster2 mirror image_13 snapshot 2023-10-11T10:56:05.501 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:56:05.501 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:56:05.501 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_13 2023-10-11T10:56:05.501 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local mode=snapshot 2023-10-11T10:56:05.502 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_13 snapshot 2023-10-11T10:56:06.365 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:Mirroring enabled 2023-10-11T10:56:06.375 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 info mirror/image_13 2023-10-11T10:56:06.414 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:rbd image 'image_13': 2023-10-11T10:56:06.415 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: size 128 MiB in 32 objects 2023-10-11T10:56:06.415 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: order 22 (4 MiB objects) 2023-10-11T10:56:06.415 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: snapshot_count: 1 2023-10-11T10:56:06.415 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: id: 1295efd9b032 2023-10-11T10:56:06.415 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: block_name_prefix: rbd_data.1295efd9b032 2023-10-11T10:56:06.416 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: format: 2 2023-10-11T10:56:06.416 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-10-11T10:56:06.416 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: op_features: 2023-10-11T10:56:06.416 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: flags: 2023-10-11T10:56:06.416 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: create_timestamp: Wed Oct 11 10:56:05 2023 2023-10-11T10:56:06.416 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: access_timestamp: Wed Oct 11 10:56:05 2023 2023-10-11T10:56:06.417 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: modify_timestamp: Wed Oct 11 10:56:05 2023 2023-10-11T10:56:06.417 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring state: enabled 2023-10-11T10:56:06.417 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring mode: snapshot 2023-10-11T10:56:06.417 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring global id: bea67517-2558-4f8d-8c9b-71c5be4fe773 2023-10-11T10:56:06.417 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring primary: true 2023-10-11T10:56:06.418 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n '' ']' 2023-10-11T10:56:06.419 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ stress_write_image cluster2 mirror image_13 2023-10-11T10:56:06.419 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:56:06.419 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:56:06.419 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_13 2023-10-11T10:56:06.419 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-10-11T10:56:06.420 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local duration=7 2023-10-11T10:56:06.420 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ set +e 2023-10-11T10:56:06.420 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_13 --debug-rbd=20 --debug-journaler=20 2023-10-11T10:56:06.747 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:100 IOs, 88388608 bytes 2023-10-11T10:56:06.969 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:200 IOs, 158419968 bytes 2023-10-11T10:56:07.060 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:300 IOs, 159038464 bytes 2023-10-11T10:56:07.266 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:400 IOs, 225054720 bytes 2023-10-11T10:56:07.451 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:500 IOs, 288336896 bytes 2023-10-11T10:56:07.624 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:600 IOs, 311709696 bytes 2023-10-11T10:56:07.822 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:700 IOs, 371484672 bytes 2023-10-11T10:56:08.006 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:800 IOs, 414132224 bytes 2023-10-11T10:56:08.137 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:900 IOs, 438733824 bytes 2023-10-11T10:56:08.308 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1000 IOs, 483886080 bytes 2023-10-11T10:56:08.511 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1100 IOs, 551789568 bytes 2023-10-11T10:56:08.654 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1200 IOs, 567964672 bytes 2023-10-11T10:56:08.773 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1300 IOs, 578913280 bytes 2023-10-11T10:56:08.994 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1400 IOs, 647996416 bytes 2023-10-11T10:56:09.186 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1500 IOs, 709170176 bytes 2023-10-11T10:56:09.418 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1600 IOs, 770610176 bytes 2023-10-11T10:56:09.603 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1700 IOs, 813172736 bytes 2023-10-11T10:56:09.762 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1800 IOs, 848142336 bytes 2023-10-11T10:56:09.933 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1900 IOs, 900141056 bytes 2023-10-11T10:56:10.092 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2000 IOs, 930393088 bytes 2023-10-11T10:56:10.291 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2100 IOs, 980661248 bytes 2023-10-11T10:56:10.441 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2200 IOs, 1026478080 bytes 2023-10-11T10:56:10.643 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2300 IOs, 1058789376 bytes 2023-10-11T10:56:10.806 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2400 IOs, 1091042304 bytes 2023-10-11T10:56:11.022 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2500 IOs, 1162750976 bytes 2023-10-11T10:56:11.203 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2600 IOs, 1213593600 bytes 2023-10-11T10:56:11.395 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2700 IOs, 1253000192 bytes 2023-10-11T10:56:11.585 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2800 IOs, 1301118976 bytes 2023-10-11T10:56:11.799 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2900 IOs, 1372333056 bytes 2023-10-11T10:56:12.038 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3000 IOs, 1443350528 bytes 2023-10-11T10:56:12.351 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3100 IOs, 1538544640 bytes 2023-10-11T10:56:12.488 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3200 IOs, 1563076608 bytes 2023-10-11T10:56:12.651 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3300 IOs, 1598799872 bytes 2023-10-11T10:56:12.748 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3400 IOs, 1599411200 bytes 2023-10-11T10:56:12.900 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3500 IOs, 1626875904 bytes 2023-10-11T10:56:13.040 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3600 IOs, 1652062208 bytes 2023-10-11T10:56:13.166 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3700 IOs, 1667790848 bytes 2023-10-11T10:56:13.314 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3800 IOs, 1700046848 bytes 2023-10-11T10:56:13.426 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ error_code=124 2023-10-11T10:56:13.426 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ set -e 2023-10-11T10:56:13.426 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' 124 -eq 124 ']' 2023-10-11T10:56:13.427 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T10:56:13.427 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T10:56:13.427 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_14 2023-10-11T10:56:13.427 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ create_image_and_enable_mirror cluster2 mirror image_14 snapshot 128M 2023-10-11T10:56:13.427 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:56:13.428 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:56:13.428 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:56:13.428 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:56:13.428 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_14 2023-10-11T10:56:13.428 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:56:13.428 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local mode=snapshot 2023-10-11T10:56:13.429 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n snapshot ']' 2023-10-11T10:56:13.429 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:56:13.429 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ create_image cluster2 mirror image_14 128M 2023-10-11T10:56:13.429 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:56:13.429 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:56:13.430 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:56:13.430 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:56:13.430 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_14 2023-10-11T10:56:13.430 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:56:13.430 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local size=128 2023-10-11T10:56:13.431 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n 128M ']' 2023-10-11T10:56:13.431 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ size=128M 2023-10-11T10:56:13.431 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:56:13.431 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_14 2023-10-11T10:56:13.471 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' image = image ']' 2023-10-11T10:56:13.471 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ enable_mirror cluster2 mirror image_14 snapshot 2023-10-11T10:56:13.471 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:56:13.471 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:56:13.472 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_14 2023-10-11T10:56:13.472 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local mode=snapshot 2023-10-11T10:56:13.472 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_14 snapshot 2023-10-11T10:56:14.354 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:Mirroring enabled 2023-10-11T10:56:14.365 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 info mirror/image_14 2023-10-11T10:56:14.401 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:rbd image 'image_14': 2023-10-11T10:56:14.402 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: size 128 MiB in 32 objects 2023-10-11T10:56:14.402 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: order 22 (4 MiB objects) 2023-10-11T10:56:14.402 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: snapshot_count: 1 2023-10-11T10:56:14.402 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: id: 12a12f278690 2023-10-11T10:56:14.402 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: block_name_prefix: rbd_data.12a12f278690 2023-10-11T10:56:14.403 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: format: 2 2023-10-11T10:56:14.403 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-10-11T10:56:14.403 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: op_features: 2023-10-11T10:56:14.403 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: flags: 2023-10-11T10:56:14.403 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: create_timestamp: Wed Oct 11 10:56:13 2023 2023-10-11T10:56:14.404 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: access_timestamp: Wed Oct 11 10:56:13 2023 2023-10-11T10:56:14.404 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: modify_timestamp: Wed Oct 11 10:56:13 2023 2023-10-11T10:56:14.404 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring state: enabled 2023-10-11T10:56:14.404 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring mode: snapshot 2023-10-11T10:56:14.404 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring global id: cc29ef48-4a54-4049-9722-bc6c6b7caf39 2023-10-11T10:56:14.404 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring primary: true 2023-10-11T10:56:14.405 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n '' ']' 2023-10-11T10:56:14.405 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ stress_write_image cluster2 mirror image_14 2023-10-11T10:56:14.406 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:56:14.406 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:56:14.406 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_14 2023-10-11T10:56:14.406 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-10-11T10:56:14.407 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local duration=12 2023-10-11T10:56:14.407 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ set +e 2023-10-11T10:56:14.407 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_14 --debug-rbd=20 --debug-journaler=20 2023-10-11T10:56:14.655 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:100 IOs, 44457984 bytes 2023-10-11T10:56:14.743 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:200 IOs, 45065216 bytes 2023-10-11T10:56:14.947 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:300 IOs, 88997888 bytes 2023-10-11T10:56:15.090 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:400 IOs, 115929088 bytes 2023-10-11T10:56:15.288 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:500 IOs, 170319872 bytes 2023-10-11T10:56:15.471 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:600 IOs, 219211776 bytes 2023-10-11T10:56:15.590 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:700 IOs, 233462784 bytes 2023-10-11T10:56:15.809 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:800 IOs, 294465536 bytes 2023-10-11T10:56:15.950 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:900 IOs, 324102144 bytes 2023-10-11T10:56:16.129 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1000 IOs, 381359104 bytes 2023-10-11T10:56:16.305 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1100 IOs, 401008640 bytes 2023-10-11T10:56:16.461 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1200 IOs, 437165056 bytes 2023-10-11T10:56:16.593 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1300 IOs, 460779520 bytes 2023-10-11T10:56:16.846 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1400 IOs, 551677952 bytes 2023-10-11T10:56:17.016 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1500 IOs, 574844928 bytes 2023-10-11T10:56:17.285 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1600 IOs, 656256000 bytes 2023-10-11T10:56:17.384 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1700 IOs, 656896000 bytes 2023-10-11T10:56:17.516 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1800 IOs, 681154560 bytes 2023-10-11T10:56:17.775 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1900 IOs, 760331264 bytes 2023-10-11T10:56:18.060 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2000 IOs, 854524928 bytes 2023-10-11T10:56:18.278 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2100 IOs, 921120768 bytes 2023-10-11T10:56:18.368 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2200 IOs, 922087424 bytes 2023-10-11T10:56:18.521 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2300 IOs, 954968064 bytes 2023-10-11T10:56:18.688 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2400 IOs, 993867776 bytes 2023-10-11T10:56:18.889 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2500 IOs, 1059088384 bytes 2023-10-11T10:56:19.043 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2600 IOs, 1089486848 bytes 2023-10-11T10:56:19.242 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2700 IOs, 1135723520 bytes 2023-10-11T10:56:19.348 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2800 IOs, 1145867264 bytes 2023-10-11T10:56:19.547 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2900 IOs, 1205938176 bytes 2023-10-11T10:56:19.823 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3000 IOs, 1298487296 bytes 2023-10-11T10:56:19.962 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3100 IOs, 1328071680 bytes 2023-10-11T10:56:20.144 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3200 IOs, 1367330816 bytes 2023-10-11T10:56:20.278 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3300 IOs, 1386699776 bytes 2023-10-11T10:56:20.516 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3400 IOs, 1457419264 bytes 2023-10-11T10:56:20.701 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3500 IOs, 1523943424 bytes 2023-10-11T10:56:20.894 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3600 IOs, 1579334656 bytes 2023-10-11T10:56:21.105 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3700 IOs, 1621295104 bytes 2023-10-11T10:56:21.321 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3800 IOs, 1683926016 bytes 2023-10-11T10:56:21.444 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3900 IOs, 1700528128 bytes 2023-10-11T10:56:21.533 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4000 IOs, 1704243200 bytes 2023-10-11T10:56:21.631 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4100 IOs, 1705262080 bytes 2023-10-11T10:56:21.780 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4200 IOs, 1738219520 bytes 2023-10-11T10:56:21.971 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4300 IOs, 1769099264 bytes 2023-10-11T10:56:22.075 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4400 IOs, 1775079424 bytes 2023-10-11T10:56:22.225 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4500 IOs, 1812828160 bytes 2023-10-11T10:56:22.340 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4600 IOs, 1821319168 bytes 2023-10-11T10:56:22.542 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4700 IOs, 1896922112 bytes 2023-10-11T10:56:22.775 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4800 IOs, 1966394368 bytes 2023-10-11T10:56:22.962 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4900 IOs, 1994906624 bytes 2023-10-11T10:56:23.197 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5000 IOs, 2061340672 bytes 2023-10-11T10:56:23.322 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5100 IOs, 2083287040 bytes 2023-10-11T10:56:23.647 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5200 IOs, 2204656640 bytes 2023-10-11T10:56:23.835 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5300 IOs, 2273173504 bytes 2023-10-11T10:56:24.043 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5400 IOs, 2305812480 bytes 2023-10-11T10:56:24.254 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5500 IOs, 2362968064 bytes 2023-10-11T10:56:24.427 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5600 IOs, 2414324736 bytes 2023-10-11T10:56:24.604 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5700 IOs, 2454964224 bytes 2023-10-11T10:56:24.738 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5800 IOs, 2471948288 bytes 2023-10-11T10:56:24.922 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5900 IOs, 2516006912 bytes 2023-10-11T10:56:25.052 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6000 IOs, 2529842176 bytes 2023-10-11T10:56:25.205 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6100 IOs, 2567545856 bytes 2023-10-11T10:56:25.529 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6200 IOs, 2678470656 bytes 2023-10-11T10:56:25.787 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6300 IOs, 2748471296 bytes 2023-10-11T10:56:25.952 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6400 IOs, 2792901632 bytes 2023-10-11T10:56:26.061 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6500 IOs, 2796065792 bytes 2023-10-11T10:56:26.165 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6600 IOs, 2800291840 bytes 2023-10-11T10:56:26.265 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6700 IOs, 2800887808 bytes 2023-10-11T10:56:26.393 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6800 IOs, 2821314560 bytes 2023-10-11T10:56:26.416 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ error_code=124 2023-10-11T10:56:26.416 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ set -e 2023-10-11T10:56:26.416 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' 124 -eq 124 ']' 2023-10-11T10:56:26.416 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T10:56:26.416 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T10:56:26.417 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_15 2023-10-11T10:56:26.417 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ create_image_and_enable_mirror cluster2 mirror image_15 snapshot 128M 2023-10-11T10:56:26.417 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:56:26.417 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:56:26.417 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:56:26.417 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:56:26.418 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_15 2023-10-11T10:56:26.418 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:56:26.418 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local mode=snapshot 2023-10-11T10:56:26.418 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n snapshot ']' 2023-10-11T10:56:26.418 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:56:26.419 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ create_image cluster2 mirror image_15 128M 2023-10-11T10:56:26.419 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:56:26.419 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:56:26.419 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:56:26.419 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:56:26.420 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_15 2023-10-11T10:56:26.420 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:56:26.420 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local size=128 2023-10-11T10:56:26.420 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n 128M ']' 2023-10-11T10:56:26.420 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ size=128M 2023-10-11T10:56:26.420 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:56:26.421 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_15 2023-10-11T10:56:26.466 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' image = image ']' 2023-10-11T10:56:26.466 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ enable_mirror cluster2 mirror image_15 snapshot 2023-10-11T10:56:26.466 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:56:26.466 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:56:26.467 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_15 2023-10-11T10:56:26.467 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local mode=snapshot 2023-10-11T10:56:26.467 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_15 snapshot 2023-10-11T10:56:27.443 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:Mirroring enabled 2023-10-11T10:56:27.452 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 info mirror/image_15 2023-10-11T10:56:27.488 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:rbd image 'image_15': 2023-10-11T10:56:27.489 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: size 128 MiB in 32 objects 2023-10-11T10:56:27.489 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: order 22 (4 MiB objects) 2023-10-11T10:56:27.489 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: snapshot_count: 1 2023-10-11T10:56:27.489 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: id: 12adf8fde845 2023-10-11T10:56:27.489 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: block_name_prefix: rbd_data.12adf8fde845 2023-10-11T10:56:27.490 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: format: 2 2023-10-11T10:56:27.490 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-10-11T10:56:27.490 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: op_features: 2023-10-11T10:56:27.490 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: flags: 2023-10-11T10:56:27.490 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: create_timestamp: Wed Oct 11 10:56:26 2023 2023-10-11T10:56:27.491 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: access_timestamp: Wed Oct 11 10:56:26 2023 2023-10-11T10:56:27.491 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: modify_timestamp: Wed Oct 11 10:56:26 2023 2023-10-11T10:56:27.491 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring state: enabled 2023-10-11T10:56:27.491 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring mode: snapshot 2023-10-11T10:56:27.491 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring global id: 7f696646-7f90-4d09-a66b-990e8fbb123e 2023-10-11T10:56:27.491 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring primary: true 2023-10-11T10:56:27.492 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n '' ']' 2023-10-11T10:56:27.492 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ stress_write_image cluster2 mirror image_15 2023-10-11T10:56:27.492 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:56:27.493 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:56:27.493 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_15 2023-10-11T10:56:27.493 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-10-11T10:56:27.493 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local duration=14 2023-10-11T10:56:27.494 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ set +e 2023-10-11T10:56:27.494 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_15 --debug-rbd=20 --debug-journaler=20 2023-10-11T10:56:27.760 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:100 IOs, 58411008 bytes 2023-10-11T10:56:27.947 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:200 IOs, 102001664 bytes 2023-10-11T10:56:28.127 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:300 IOs, 161800192 bytes 2023-10-11T10:56:28.268 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:400 IOs, 176338944 bytes 2023-10-11T10:56:28.392 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:500 IOs, 191099904 bytes 2023-10-11T10:56:28.558 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:600 IOs, 228026368 bytes 2023-10-11T10:56:28.690 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:700 IOs, 252896256 bytes 2023-10-11T10:56:28.916 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:800 IOs, 330554368 bytes 2023-10-11T10:56:29.127 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:900 IOs, 408307712 bytes 2023-10-11T10:56:29.352 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1000 IOs, 456479744 bytes 2023-10-11T10:56:29.667 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1100 IOs, 570115072 bytes 2023-10-11T10:56:29.911 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1200 IOs, 653379584 bytes 2023-10-11T10:56:30.009 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1300 IOs, 659038208 bytes 2023-10-11T10:56:30.148 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1400 IOs, 684377088 bytes 2023-10-11T10:56:30.263 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1500 IOs, 698444800 bytes 2023-10-11T10:56:30.464 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1600 IOs, 754024448 bytes 2023-10-11T10:56:30.689 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1700 IOs, 818619392 bytes 2023-10-11T10:56:30.868 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1800 IOs, 864331776 bytes 2023-10-11T10:56:31.041 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1900 IOs, 898778112 bytes 2023-10-11T10:56:31.175 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2000 IOs, 926301184 bytes 2023-10-11T10:56:31.412 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2100 IOs, 1000524800 bytes 2023-10-11T10:56:31.609 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2200 IOs, 1062719488 bytes 2023-10-11T10:56:31.715 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2300 IOs, 1068843008 bytes 2023-10-11T10:56:31.861 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2400 IOs, 1098653696 bytes 2023-10-11T10:56:32.136 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2500 IOs, 1183822848 bytes 2023-10-11T10:56:32.223 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2600 IOs, 1184433152 bytes 2023-10-11T10:56:32.425 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2700 IOs, 1240269824 bytes 2023-10-11T10:56:32.614 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2800 IOs, 1289934848 bytes 2023-10-11T10:56:32.748 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2900 IOs, 1325443072 bytes 2023-10-11T10:56:33.070 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3000 IOs, 1417929728 bytes 2023-10-11T10:56:33.262 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3100 IOs, 1468911616 bytes 2023-10-11T10:56:33.368 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3200 IOs, 1472376832 bytes 2023-10-11T10:56:33.586 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3300 IOs, 1549304832 bytes 2023-10-11T10:56:33.821 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3400 IOs, 1613928448 bytes 2023-10-11T10:56:33.908 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3500 IOs, 1614529536 bytes 2023-10-11T10:56:34.055 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3600 IOs, 1638444032 bytes 2023-10-11T10:56:34.258 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3700 IOs, 1699869696 bytes 2023-10-11T10:56:34.432 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3800 IOs, 1747452928 bytes 2023-10-11T10:56:34.571 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3900 IOs, 1767881728 bytes 2023-10-11T10:56:34.795 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4000 IOs, 1841296384 bytes 2023-10-11T10:56:34.983 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4100 IOs, 1884358656 bytes 2023-10-11T10:56:35.266 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4200 IOs, 1984007168 bytes 2023-10-11T10:56:35.430 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4300 IOs, 2014311424 bytes 2023-10-11T10:56:35.532 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4400 IOs, 2015394816 bytes 2023-10-11T10:56:35.688 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4500 IOs, 2065691648 bytes 2023-10-11T10:56:35.812 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4600 IOs, 2066328576 bytes 2023-10-11T10:56:35.965 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4700 IOs, 2094045184 bytes 2023-10-11T10:56:36.193 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4800 IOs, 2157910016 bytes 2023-10-11T10:56:36.308 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4900 IOs, 2169235456 bytes 2023-10-11T10:56:36.531 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5000 IOs, 2234271744 bytes 2023-10-11T10:56:36.824 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5100 IOs, 2345204736 bytes 2023-10-11T10:56:36.950 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5200 IOs, 2364914688 bytes 2023-10-11T10:56:37.055 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5300 IOs, 2374478848 bytes 2023-10-11T10:56:37.241 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5400 IOs, 2434607104 bytes 2023-10-11T10:56:37.465 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5500 IOs, 2487945216 bytes 2023-10-11T10:56:37.633 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5600 IOs, 2531317760 bytes 2023-10-11T10:56:37.718 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5700 IOs, 2531910656 bytes 2023-10-11T10:56:37.812 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5800 IOs, 2534667264 bytes 2023-10-11T10:56:37.937 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5900 IOs, 2545080320 bytes 2023-10-11T10:56:38.062 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6000 IOs, 2569518080 bytes 2023-10-11T10:56:38.300 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6100 IOs, 2638714880 bytes 2023-10-11T10:56:38.600 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6200 IOs, 2771328000 bytes 2023-10-11T10:56:38.851 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6300 IOs, 2833290240 bytes 2023-10-11T10:56:39.028 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6400 IOs, 2871837696 bytes 2023-10-11T10:56:39.196 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6500 IOs, 2913343488 bytes 2023-10-11T10:56:39.471 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6600 IOs, 2997310464 bytes 2023-10-11T10:56:39.635 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6700 IOs, 3032361984 bytes 2023-10-11T10:56:39.812 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6800 IOs, 3086618624 bytes 2023-10-11T10:56:39.986 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6900 IOs, 3127830528 bytes 2023-10-11T10:56:40.146 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:7000 IOs, 3160065024 bytes 2023-10-11T10:56:40.275 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:7100 IOs, 3186723840 bytes 2023-10-11T10:56:40.497 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:7200 IOs, 3249867776 bytes 2023-10-11T10:56:40.645 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:7300 IOs, 3276301312 bytes 2023-10-11T10:56:40.879 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:7400 IOs, 3340612608 bytes 2023-10-11T10:56:40.981 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:7500 IOs, 3343183872 bytes 2023-10-11T10:56:41.178 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:7600 IOs, 3400645632 bytes 2023-10-11T10:56:41.479 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:7700 IOs, 3520717824 bytes 2023-10-11T10:56:41.503 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ error_code=124 2023-10-11T10:56:41.504 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ set -e 2023-10-11T10:56:41.504 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' 124 -eq 124 ']' 2023-10-11T10:56:41.504 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T10:56:41.504 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T10:56:41.504 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_16 2023-10-11T10:56:41.505 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ create_image_and_enable_mirror cluster2 mirror image_16 snapshot 128M 2023-10-11T10:56:41.505 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:56:41.505 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:56:41.505 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:56:41.505 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:56:41.505 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_16 2023-10-11T10:56:41.506 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:56:41.506 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local mode=snapshot 2023-10-11T10:56:41.506 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n snapshot ']' 2023-10-11T10:56:41.506 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:56:41.506 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ create_image cluster2 mirror image_16 128M 2023-10-11T10:56:41.507 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:56:41.507 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:56:41.507 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:56:41.507 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:56:41.507 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_16 2023-10-11T10:56:41.507 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:56:41.508 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local size=128 2023-10-11T10:56:41.508 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n 128M ']' 2023-10-11T10:56:41.508 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ size=128M 2023-10-11T10:56:41.508 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:56:41.508 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_16 2023-10-11T10:56:41.553 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' image = image ']' 2023-10-11T10:56:41.553 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ enable_mirror cluster2 mirror image_16 snapshot 2023-10-11T10:56:41.553 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:56:41.554 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:56:41.554 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_16 2023-10-11T10:56:41.554 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local mode=snapshot 2023-10-11T10:56:41.554 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_16 snapshot 2023-10-11T10:56:42.124 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:Mirroring enabled 2023-10-11T10:56:42.136 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 info mirror/image_16 2023-10-11T10:56:42.176 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:rbd image 'image_16': 2023-10-11T10:56:42.176 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: size 128 MiB in 32 objects 2023-10-11T10:56:42.177 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: order 22 (4 MiB objects) 2023-10-11T10:56:42.177 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: snapshot_count: 1 2023-10-11T10:56:42.177 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: id: 12b9bba45647 2023-10-11T10:56:42.177 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: block_name_prefix: rbd_data.12b9bba45647 2023-10-11T10:56:42.177 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: format: 2 2023-10-11T10:56:42.178 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-10-11T10:56:42.178 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: op_features: 2023-10-11T10:56:42.178 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: flags: 2023-10-11T10:56:42.178 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: create_timestamp: Wed Oct 11 10:56:41 2023 2023-10-11T10:56:42.178 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: access_timestamp: Wed Oct 11 10:56:41 2023 2023-10-11T10:56:42.178 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: modify_timestamp: Wed Oct 11 10:56:41 2023 2023-10-11T10:56:42.179 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring state: enabled 2023-10-11T10:56:42.179 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring mode: snapshot 2023-10-11T10:56:42.179 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring global id: 2ba7e4f0-06a3-4db0-aeee-64ab913d9bf0 2023-10-11T10:56:42.179 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring primary: true 2023-10-11T10:56:42.180 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n '' ']' 2023-10-11T10:56:42.180 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ stress_write_image cluster2 mirror image_16 2023-10-11T10:56:42.180 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:56:42.180 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:56:42.180 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_16 2023-10-11T10:56:42.181 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-10-11T10:56:42.181 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local duration=8 2023-10-11T10:56:42.182 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ set +e 2023-10-11T10:56:42.182 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_16 --debug-rbd=20 --debug-journaler=20 2023-10-11T10:56:42.423 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:100 IOs, 20041728 bytes 2023-10-11T10:56:42.650 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:200 IOs, 91899904 bytes 2023-10-11T10:56:42.803 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:300 IOs, 136163328 bytes 2023-10-11T10:56:43.064 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:400 IOs, 195835904 bytes 2023-10-11T10:56:43.232 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:500 IOs, 235880448 bytes 2023-10-11T10:56:43.423 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:600 IOs, 287675392 bytes 2023-10-11T10:56:43.599 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:700 IOs, 330505216 bytes 2023-10-11T10:56:43.759 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:800 IOs, 366794752 bytes 2023-10-11T10:56:43.938 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:900 IOs, 409654272 bytes 2023-10-11T10:56:44.032 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1000 IOs, 410270720 bytes 2023-10-11T10:56:44.251 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1100 IOs, 470968320 bytes 2023-10-11T10:56:44.481 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1200 IOs, 550476800 bytes 2023-10-11T10:56:44.694 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1300 IOs, 598220800 bytes 2023-10-11T10:56:44.841 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1400 IOs, 622965760 bytes 2023-10-11T10:56:45.004 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1500 IOs, 655618048 bytes 2023-10-11T10:56:45.141 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1600 IOs, 681308160 bytes 2023-10-11T10:56:45.255 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1700 IOs, 692678656 bytes 2023-10-11T10:56:45.477 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1800 IOs, 754409472 bytes 2023-10-11T10:56:45.737 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1900 IOs, 848769024 bytes 2023-10-11T10:56:45.892 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2000 IOs, 875116544 bytes 2023-10-11T10:56:46.022 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2100 IOs, 897511424 bytes 2023-10-11T10:56:46.260 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2200 IOs, 992535552 bytes 2023-10-11T10:56:46.488 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2300 IOs, 1040697344 bytes 2023-10-11T10:56:46.654 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2400 IOs, 1061647360 bytes 2023-10-11T10:56:46.886 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2500 IOs, 1137753088 bytes 2023-10-11T10:56:47.058 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2600 IOs, 1174854656 bytes 2023-10-11T10:56:47.270 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2700 IOs, 1226263552 bytes 2023-10-11T10:56:47.528 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2800 IOs, 1319146496 bytes 2023-10-11T10:56:47.727 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2900 IOs, 1362572288 bytes 2023-10-11T10:56:47.906 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3000 IOs, 1401216000 bytes 2023-10-11T10:56:48.043 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3100 IOs, 1421694976 bytes 2023-10-11T10:56:48.231 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3200 IOs, 1460687872 bytes 2023-10-11T10:56:48.448 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3300 IOs, 1512446976 bytes 2023-10-11T10:56:48.561 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3400 IOs, 1513469952 bytes 2023-10-11T10:56:48.734 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3500 IOs, 1549923328 bytes 2023-10-11T10:56:48.862 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3600 IOs, 1562948608 bytes 2023-10-11T10:56:49.045 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3700 IOs, 1608108032 bytes 2023-10-11T10:56:49.234 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3800 IOs, 1652776960 bytes 2023-10-11T10:56:49.421 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3900 IOs, 1701488640 bytes 2023-10-11T10:56:49.617 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4000 IOs, 1751390208 bytes 2023-10-11T10:56:49.757 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4100 IOs, 1763232768 bytes 2023-10-11T10:56:49.874 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4200 IOs, 1773171712 bytes 2023-10-11T10:56:50.056 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4300 IOs, 1826117632 bytes 2023-10-11T10:56:50.188 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ error_code=124 2023-10-11T10:56:50.188 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ set -e 2023-10-11T10:56:50.188 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' 124 -eq 124 ']' 2023-10-11T10:56:50.188 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T10:56:50.188 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T10:56:50.189 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_17 2023-10-11T10:56:50.189 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ create_image_and_enable_mirror cluster2 mirror image_17 snapshot 128M 2023-10-11T10:56:50.189 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:56:50.189 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:56:50.189 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:56:50.190 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:56:50.190 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_17 2023-10-11T10:56:50.190 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:56:50.190 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local mode=snapshot 2023-10-11T10:56:50.190 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n snapshot ']' 2023-10-11T10:56:50.191 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:56:50.191 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ create_image cluster2 mirror image_17 128M 2023-10-11T10:56:50.191 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:56:50.191 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:56:50.191 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:56:50.191 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:56:50.192 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_17 2023-10-11T10:56:50.192 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:56:50.192 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local size=128 2023-10-11T10:56:50.192 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n 128M ']' 2023-10-11T10:56:50.192 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ size=128M 2023-10-11T10:56:50.193 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:56:50.193 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_17 2023-10-11T10:56:50.237 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' image = image ']' 2023-10-11T10:56:50.238 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ enable_mirror cluster2 mirror image_17 snapshot 2023-10-11T10:56:50.238 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:56:50.238 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:56:50.238 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_17 2023-10-11T10:56:50.238 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local mode=snapshot 2023-10-11T10:56:50.239 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_17 snapshot 2023-10-11T10:56:51.118 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:Mirroring enabled 2023-10-11T10:56:51.127 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 info mirror/image_17 2023-10-11T10:56:51.164 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:rbd image 'image_17': 2023-10-11T10:56:51.164 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: size 128 MiB in 32 objects 2023-10-11T10:56:51.164 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: order 22 (4 MiB objects) 2023-10-11T10:56:51.164 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: snapshot_count: 1 2023-10-11T10:56:51.164 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: id: 12c59d440b4d 2023-10-11T10:56:51.165 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: block_name_prefix: rbd_data.12c59d440b4d 2023-10-11T10:56:51.165 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: format: 2 2023-10-11T10:56:51.165 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-10-11T10:56:51.165 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: op_features: 2023-10-11T10:56:51.165 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: flags: 2023-10-11T10:56:51.166 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: create_timestamp: Wed Oct 11 10:56:50 2023 2023-10-11T10:56:51.166 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: access_timestamp: Wed Oct 11 10:56:50 2023 2023-10-11T10:56:51.166 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: modify_timestamp: Wed Oct 11 10:56:50 2023 2023-10-11T10:56:51.166 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring state: enabled 2023-10-11T10:56:51.166 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring mode: snapshot 2023-10-11T10:56:51.167 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring global id: f3a165bf-82fd-4110-9215-0cf728d304ac 2023-10-11T10:56:51.167 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring primary: true 2023-10-11T10:56:51.168 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n '' ']' 2023-10-11T10:56:51.168 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ stress_write_image cluster2 mirror image_17 2023-10-11T10:56:51.168 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:56:51.168 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:56:51.168 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_17 2023-10-11T10:56:51.169 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-10-11T10:56:51.169 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local duration=14 2023-10-11T10:56:51.169 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ set +e 2023-10-11T10:56:51.170 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_17 --debug-rbd=20 --debug-journaler=20 2023-10-11T10:56:51.448 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:100 IOs, 57896960 bytes 2023-10-11T10:56:51.637 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:200 IOs, 103058432 bytes 2023-10-11T10:56:51.743 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:300 IOs, 105259008 bytes 2023-10-11T10:56:51.856 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:400 IOs, 131154944 bytes 2023-10-11T10:56:52.020 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:500 IOs, 150657024 bytes 2023-10-11T10:56:52.296 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:600 IOs, 242971648 bytes 2023-10-11T10:56:52.546 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:700 IOs, 324796416 bytes 2023-10-11T10:56:52.708 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:800 IOs, 359145472 bytes 2023-10-11T10:56:52.809 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:900 IOs, 376015872 bytes 2023-10-11T10:56:52.991 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1000 IOs, 407357440 bytes 2023-10-11T10:56:53.111 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1100 IOs, 417166336 bytes 2023-10-11T10:56:53.244 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1200 IOs, 444099584 bytes 2023-10-11T10:56:53.357 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1300 IOs, 447469568 bytes 2023-10-11T10:56:53.575 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1400 IOs, 504807424 bytes 2023-10-11T10:56:53.871 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1500 IOs, 636937216 bytes 2023-10-11T10:56:54.125 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1600 IOs, 693947392 bytes 2023-10-11T10:56:54.301 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1700 IOs, 737824768 bytes 2023-10-11T10:56:54.521 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1800 IOs, 800112640 bytes 2023-10-11T10:56:54.538 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.34741172797850806) > weight (0.25) 2023-10-11T10:56:54.538 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.9930373115268548) > weight (0.25) 2023-10-11T10:56:54.538 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.2 2023-10-11T10:56:54.538 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Sent signal 15 2023-10-11T10:56:54.539 INFO:tasks.rbd_mirror_thrash:waiting for 8 secs before reviving daemons 2023-10-11T10:56:54.539 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi111.stderr:2023-10-11T10:56:54.539+0000 7fb349c4a700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 (PID: 112244) UID: 1000 2023-10-11T10:56:54.923 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1900 IOs, 865156096 bytes 2023-10-11T10:56:55.065 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2000 IOs, 882833408 bytes 2023-10-11T10:56:55.220 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2100 IOs, 917816320 bytes 2023-10-11T10:56:55.499 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2200 IOs, 1015154688 bytes 2023-10-11T10:56:55.704 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2300 IOs, 1062803456 bytes 2023-10-11T10:56:55.875 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2400 IOs, 1102883840 bytes 2023-10-11T10:56:56.091 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2500 IOs, 1160933376 bytes 2023-10-11T10:56:56.494 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2600 IOs, 1314370560 bytes 2023-10-11T10:56:56.749 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2700 IOs, 1394838528 bytes 2023-10-11T10:56:56.856 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2800 IOs, 1395433472 bytes 2023-10-11T10:56:56.988 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2900 IOs, 1419788288 bytes 2023-10-11T10:56:57.190 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3000 IOs, 1471775744 bytes 2023-10-11T10:56:57.347 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3100 IOs, 1502167040 bytes 2023-10-11T10:56:57.441 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3200 IOs, 1502794752 bytes 2023-10-11T10:56:57.647 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3300 IOs, 1556926464 bytes 2023-10-11T10:56:57.827 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3400 IOs, 1598152704 bytes 2023-10-11T10:56:58.078 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3500 IOs, 1673400320 bytes 2023-10-11T10:56:58.280 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3600 IOs, 1729952768 bytes 2023-10-11T10:56:58.419 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3700 IOs, 1750559744 bytes 2023-10-11T10:56:58.568 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3800 IOs, 1764322304 bytes 2023-10-11T10:56:58.774 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3900 IOs, 1817562112 bytes 2023-10-11T10:56:58.916 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4000 IOs, 1843067904 bytes 2023-10-11T10:56:59.182 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4100 IOs, 1939005440 bytes 2023-10-11T10:56:59.311 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4200 IOs, 1954631680 bytes 2023-10-11T10:56:59.442 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4300 IOs, 1980051456 bytes 2023-10-11T10:56:59.691 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4400 IOs, 2045882368 bytes 2023-10-11T10:56:59.832 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4500 IOs, 2070125568 bytes 2023-10-11T10:56:59.990 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4600 IOs, 2104038400 bytes 2023-10-11T10:57:00.179 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4700 IOs, 2149859328 bytes 2023-10-11T10:57:00.359 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4800 IOs, 2194070528 bytes 2023-10-11T10:57:00.650 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4900 IOs, 2277285888 bytes 2023-10-11T10:57:00.755 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5000 IOs, 2280771584 bytes 2023-10-11T10:57:00.858 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5100 IOs, 2295758848 bytes 2023-10-11T10:57:00.977 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5200 IOs, 2305932288 bytes 2023-10-11T10:57:01.171 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5300 IOs, 2350409728 bytes 2023-10-11T10:57:01.307 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5400 IOs, 2368013312 bytes 2023-10-11T10:57:01.473 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5500 IOs, 2398366720 bytes 2023-10-11T10:57:01.576 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5600 IOs, 2398979072 bytes 2023-10-11T10:57:01.679 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5700 IOs, 2399586304 bytes 2023-10-11T10:57:01.843 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5800 IOs, 2436125696 bytes 2023-10-11T10:57:01.980 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5900 IOs, 2476960768 bytes 2023-10-11T10:57:02.158 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6000 IOs, 2499171328 bytes 2023-10-11T10:57:02.294 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6100 IOs, 2536982528 bytes 2023-10-11T10:57:02.525 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6200 IOs, 2594333696 bytes 2023-10-11T10:57:02.540 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.2 2023-10-11T10:57:02.540 INFO:teuthology.orchestra.run:waiting for 600 2023-10-11T10:57:02.541 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.2 2023-10-11T10:57:02.541 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Restarting daemon 2023-10-11T10:57:02.542 DEBUG:teuthology.orchestra.run.smithi111:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 2023-10-11T10:57:02.545 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Started 2023-10-11T10:57:02.545 INFO:tasks.rbd_mirror_thrash:waiting for 61 secs before thrashing 2023-10-11T10:57:02.674 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6300 IOs, 2616077312 bytes 2023-10-11T10:57:02.853 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6400 IOs, 2656925696 bytes 2023-10-11T10:57:03.110 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6500 IOs, 2738104320 bytes 2023-10-11T10:57:03.285 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6600 IOs, 2774020096 bytes 2023-10-11T10:57:03.410 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6700 IOs, 2789615616 bytes 2023-10-11T10:57:03.519 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6800 IOs, 2793438208 bytes 2023-10-11T10:57:03.762 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6900 IOs, 2869778432 bytes 2023-10-11T10:57:03.868 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:7000 IOs, 2878063616 bytes 2023-10-11T10:57:04.085 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:7100 IOs, 2929584128 bytes 2023-10-11T10:57:04.213 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:7200 IOs, 2950160384 bytes 2023-10-11T10:57:04.345 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:7300 IOs, 2973519872 bytes 2023-10-11T10:57:04.617 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:7400 IOs, 3063119872 bytes 2023-10-11T10:57:04.767 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:7500 IOs, 3075577856 bytes 2023-10-11T10:57:04.885 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:7600 IOs, 3081423872 bytes 2023-10-11T10:57:05.140 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:7700 IOs, 3155999744 bytes 2023-10-11T10:57:05.179 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ error_code=124 2023-10-11T10:57:05.179 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ set -e 2023-10-11T10:57:05.179 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' 124 -eq 124 ']' 2023-10-11T10:57:05.180 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T10:57:05.180 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T10:57:05.180 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_18 2023-10-11T10:57:05.180 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ create_image_and_enable_mirror cluster2 mirror image_18 snapshot 128M 2023-10-11T10:57:05.180 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:57:05.181 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:57:05.181 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:57:05.181 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:57:05.181 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_18 2023-10-11T10:57:05.181 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:57:05.182 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local mode=snapshot 2023-10-11T10:57:05.182 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n snapshot ']' 2023-10-11T10:57:05.182 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:57:05.182 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ create_image cluster2 mirror image_18 128M 2023-10-11T10:57:05.182 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:57:05.183 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:57:05.183 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:57:05.183 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:57:05.183 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_18 2023-10-11T10:57:05.183 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:57:05.183 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local size=128 2023-10-11T10:57:05.184 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n 128M ']' 2023-10-11T10:57:05.184 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ size=128M 2023-10-11T10:57:05.184 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:57:05.184 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_18 2023-10-11T10:57:05.228 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' image = image ']' 2023-10-11T10:57:05.228 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ enable_mirror cluster2 mirror image_18 snapshot 2023-10-11T10:57:05.228 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:57:05.229 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:57:05.229 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_18 2023-10-11T10:57:05.229 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local mode=snapshot 2023-10-11T10:57:05.229 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_18 snapshot 2023-10-11T10:57:06.128 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:Mirroring enabled 2023-10-11T10:57:06.137 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 info mirror/image_18 2023-10-11T10:57:06.173 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:rbd image 'image_18': 2023-10-11T10:57:06.174 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: size 128 MiB in 32 objects 2023-10-11T10:57:06.174 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: order 22 (4 MiB objects) 2023-10-11T10:57:06.174 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: snapshot_count: 1 2023-10-11T10:57:06.174 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: id: 12d6f15677bd 2023-10-11T10:57:06.174 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: block_name_prefix: rbd_data.12d6f15677bd 2023-10-11T10:57:06.175 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: format: 2 2023-10-11T10:57:06.175 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-10-11T10:57:06.175 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: op_features: 2023-10-11T10:57:06.175 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: flags: 2023-10-11T10:57:06.175 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: create_timestamp: Wed Oct 11 10:57:05 2023 2023-10-11T10:57:06.175 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: access_timestamp: Wed Oct 11 10:57:05 2023 2023-10-11T10:57:06.176 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: modify_timestamp: Wed Oct 11 10:57:05 2023 2023-10-11T10:57:06.176 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring state: enabled 2023-10-11T10:57:06.176 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring mode: snapshot 2023-10-11T10:57:06.176 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring global id: 1526990d-f216-41c2-accc-30ee745ab7bd 2023-10-11T10:57:06.176 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring primary: true 2023-10-11T10:57:06.177 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n '' ']' 2023-10-11T10:57:06.177 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ stress_write_image cluster2 mirror image_18 2023-10-11T10:57:06.177 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:57:06.178 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:57:06.178 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_18 2023-10-11T10:57:06.178 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-10-11T10:57:06.179 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local duration=14 2023-10-11T10:57:06.179 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ set +e 2023-10-11T10:57:06.179 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_18 --debug-rbd=20 --debug-journaler=20 2023-10-11T10:57:06.454 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:100 IOs, 53573632 bytes 2023-10-11T10:57:06.654 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:200 IOs, 103051264 bytes 2023-10-11T10:57:06.913 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:300 IOs, 190957568 bytes 2023-10-11T10:57:07.076 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:400 IOs, 231426048 bytes 2023-10-11T10:57:07.229 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:500 IOs, 258722816 bytes 2023-10-11T10:57:07.340 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:600 IOs, 270339072 bytes 2023-10-11T10:57:07.456 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:700 IOs, 286273536 bytes 2023-10-11T10:57:07.571 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:800 IOs, 299984896 bytes 2023-10-11T10:57:07.834 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:900 IOs, 375273472 bytes 2023-10-11T10:57:08.030 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1000 IOs, 436417536 bytes 2023-10-11T10:57:08.181 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1100 IOs, 490248192 bytes 2023-10-11T10:57:08.413 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1200 IOs, 538355712 bytes 2023-10-11T10:57:08.516 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1300 IOs, 549766144 bytes 2023-10-11T10:57:08.764 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1400 IOs, 612875264 bytes 2023-10-11T10:57:08.961 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1500 IOs, 667776000 bytes 2023-10-11T10:57:09.138 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1600 IOs, 712705024 bytes 2023-10-11T10:57:09.282 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1700 IOs, 746872832 bytes 2023-10-11T10:57:09.553 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1800 IOs, 827324416 bytes 2023-10-11T10:57:09.657 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1900 IOs, 828894208 bytes 2023-10-11T10:57:09.762 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2000 IOs, 829501440 bytes 2023-10-11T10:57:09.947 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2100 IOs, 895249408 bytes 2023-10-11T10:57:10.224 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2200 IOs, 961530880 bytes 2023-10-11T10:57:10.386 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2300 IOs, 995941376 bytes 2023-10-11T10:57:10.563 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2400 IOs, 1036626944 bytes 2023-10-11T10:57:10.670 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2500 IOs, 1037260800 bytes 2023-10-11T10:57:10.800 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2600 IOs, 1052668928 bytes 2023-10-11T10:57:10.959 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2700 IOs, 1088219136 bytes 2023-10-11T10:57:11.152 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2800 IOs, 1135864832 bytes 2023-10-11T10:57:11.321 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2900 IOs, 1191592960 bytes 2023-10-11T10:57:11.650 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3000 IOs, 1316209664 bytes 2023-10-11T10:57:11.814 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3100 IOs, 1335654400 bytes 2023-10-11T10:57:11.944 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3200 IOs, 1347007488 bytes 2023-10-11T10:57:12.043 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3300 IOs, 1351586816 bytes 2023-10-11T10:57:12.149 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3400 IOs, 1365158912 bytes 2023-10-11T10:57:12.277 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3500 IOs, 1391097856 bytes 2023-10-11T10:57:12.477 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3600 IOs, 1450292224 bytes 2023-10-11T10:57:12.623 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3700 IOs, 1473325056 bytes 2023-10-11T10:57:12.954 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3800 IOs, 1564995584 bytes 2023-10-11T10:57:13.160 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3900 IOs, 1615552512 bytes 2023-10-11T10:57:13.324 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4000 IOs, 1639975936 bytes 2023-10-11T10:57:13.445 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4100 IOs, 1654886400 bytes 2023-10-11T10:57:13.632 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4200 IOs, 1694343168 bytes 2023-10-11T10:57:13.767 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4300 IOs, 1718957056 bytes 2023-10-11T10:57:13.905 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4400 IOs, 1742258176 bytes 2023-10-11T10:57:14.040 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4500 IOs, 1763502080 bytes 2023-10-11T10:57:14.185 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4600 IOs, 1785338880 bytes 2023-10-11T10:57:14.395 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4700 IOs, 1841965056 bytes 2023-10-11T10:57:14.604 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4800 IOs, 1921499136 bytes 2023-10-11T10:57:14.819 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4900 IOs, 1963858944 bytes 2023-10-11T10:57:14.939 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5000 IOs, 1983774720 bytes 2023-10-11T10:57:15.207 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5100 IOs, 2065119232 bytes 2023-10-11T10:57:15.353 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5200 IOs, 2081945600 bytes 2023-10-11T10:57:15.493 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5300 IOs, 2105835520 bytes 2023-10-11T10:57:15.674 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5400 IOs, 2149153792 bytes 2023-10-11T10:57:15.798 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5500 IOs, 2165516288 bytes 2023-10-11T10:57:15.896 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5600 IOs, 2187624448 bytes 2023-10-11T10:57:16.166 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5700 IOs, 2262592512 bytes 2023-10-11T10:57:16.337 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5800 IOs, 2308620288 bytes 2023-10-11T10:57:16.491 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5900 IOs, 2332750848 bytes 2023-10-11T10:57:16.727 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6000 IOs, 2404893696 bytes 2023-10-11T10:57:16.926 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6100 IOs, 2481045504 bytes 2023-10-11T10:57:17.153 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6200 IOs, 2481652736 bytes 2023-10-11T10:57:17.374 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6300 IOs, 2546439168 bytes 2023-10-11T10:57:17.484 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6400 IOs, 2566691840 bytes 2023-10-11T10:57:17.739 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6500 IOs, 2635720704 bytes 2023-10-11T10:57:17.918 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6600 IOs, 2674934784 bytes 2023-10-11T10:57:18.261 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6700 IOs, 2788142080 bytes 2023-10-11T10:57:18.509 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6800 IOs, 2865012736 bytes 2023-10-11T10:57:18.539 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2023-10-11T10:57:18.540 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2023-10-11T10:57:18.540 INFO:tasks.rbd_mirror_thrash:waiting for 4 secs before reviving daemons 2023-10-11T10:57:18.541 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi111.stderr:2023-10-11T10:57:18.539+0000 7f36bdde7700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 106641) UID: 1000 2023-10-11T10:57:18.655 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6900 IOs, 2892416000 bytes 2023-10-11T10:57:18.828 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:7000 IOs, 2924800000 bytes 2023-10-11T10:57:18.969 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:7100 IOs, 2943269888 bytes 2023-10-11T10:57:19.245 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:7200 IOs, 3035057152 bytes 2023-10-11T10:57:19.352 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:7300 IOs, 3040666624 bytes 2023-10-11T10:57:19.543 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:7400 IOs, 3108619264 bytes 2023-10-11T10:57:19.783 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:7500 IOs, 3154825216 bytes 2023-10-11T10:57:19.882 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:7600 IOs, 3155424256 bytes 2023-10-11T10:57:20.081 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:7700 IOs, 3208695808 bytes 2023-10-11T10:57:20.185 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ error_code=124 2023-10-11T10:57:20.185 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ set -e 2023-10-11T10:57:20.185 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' 124 -eq 124 ']' 2023-10-11T10:57:20.185 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T10:57:20.186 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T10:57:20.186 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_19 2023-10-11T10:57:20.186 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ create_image_and_enable_mirror cluster2 mirror image_19 snapshot 128M 2023-10-11T10:57:20.186 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:57:20.186 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:57:20.187 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:57:20.187 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:57:20.187 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_19 2023-10-11T10:57:20.187 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:57:20.187 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local mode=snapshot 2023-10-11T10:57:20.187 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n snapshot ']' 2023-10-11T10:57:20.188 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:57:20.188 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ create_image cluster2 mirror image_19 128M 2023-10-11T10:57:20.188 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:57:20.188 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:57:20.188 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:57:20.188 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:57:20.189 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_19 2023-10-11T10:57:20.189 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:57:20.189 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local size=128 2023-10-11T10:57:20.189 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n 128M ']' 2023-10-11T10:57:20.189 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ size=128M 2023-10-11T10:57:20.190 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:57:20.190 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_19 2023-10-11T10:57:20.232 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' image = image ']' 2023-10-11T10:57:20.232 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ enable_mirror cluster2 mirror image_19 snapshot 2023-10-11T10:57:20.232 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:57:20.232 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:57:20.233 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_19 2023-10-11T10:57:20.233 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local mode=snapshot 2023-10-11T10:57:20.233 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_19 snapshot 2023-10-11T10:57:21.155 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:Mirroring enabled 2023-10-11T10:57:21.166 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 info mirror/image_19 2023-10-11T10:57:21.205 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:rbd image 'image_19': 2023-10-11T10:57:21.206 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: size 128 MiB in 32 objects 2023-10-11T10:57:21.206 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: order 22 (4 MiB objects) 2023-10-11T10:57:21.206 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: snapshot_count: 1 2023-10-11T10:57:21.206 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: id: 12e2212c87e3 2023-10-11T10:57:21.207 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: block_name_prefix: rbd_data.12e2212c87e3 2023-10-11T10:57:21.207 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: format: 2 2023-10-11T10:57:21.207 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-10-11T10:57:21.207 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: op_features: 2023-10-11T10:57:21.207 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: flags: 2023-10-11T10:57:21.207 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: create_timestamp: Wed Oct 11 10:57:20 2023 2023-10-11T10:57:21.208 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: access_timestamp: Wed Oct 11 10:57:20 2023 2023-10-11T10:57:21.208 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: modify_timestamp: Wed Oct 11 10:57:20 2023 2023-10-11T10:57:21.208 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring state: enabled 2023-10-11T10:57:21.208 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring mode: snapshot 2023-10-11T10:57:21.208 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring global id: 0f059703-71ab-45c9-bae3-1fd34ca31402 2023-10-11T10:57:21.209 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring primary: true 2023-10-11T10:57:21.209 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n '' ']' 2023-10-11T10:57:21.210 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ stress_write_image cluster2 mirror image_19 2023-10-11T10:57:21.210 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:57:21.210 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:57:21.210 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_19 2023-10-11T10:57:21.211 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-10-11T10:57:21.211 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local duration=6 2023-10-11T10:57:21.211 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ set +e 2023-10-11T10:57:21.212 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_19 --debug-rbd=20 --debug-journaler=20 2023-10-11T10:57:21.616 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:100 IOs, 120006656 bytes 2023-10-11T10:57:21.743 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:200 IOs, 131835904 bytes 2023-10-11T10:57:21.923 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:300 IOs, 177299456 bytes 2023-10-11T10:57:22.101 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:400 IOs, 222533632 bytes 2023-10-11T10:57:22.278 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:500 IOs, 262110208 bytes 2023-10-11T10:57:22.400 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:600 IOs, 291238912 bytes 2023-10-11T10:57:22.542 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2023-10-11T10:57:22.542 INFO:teuthology.orchestra.run:waiting for 600 2023-10-11T10:57:22.542 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2023-10-11T10:57:22.542 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2023-10-11T10:57:22.543 DEBUG:teuthology.orchestra.run.smithi111:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2023-10-11T10:57:22.545 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2023-10-11T10:57:22.545 INFO:tasks.rbd_mirror_thrash:waiting for 102 secs before thrashing 2023-10-11T10:57:22.649 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:700 IOs, 353486848 bytes 2023-10-11T10:57:22.808 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:800 IOs, 381209600 bytes 2023-10-11T10:57:23.088 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:900 IOs, 469306368 bytes 2023-10-11T10:57:23.262 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1000 IOs, 511542272 bytes 2023-10-11T10:57:23.455 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1100 IOs, 568359936 bytes 2023-10-11T10:57:23.672 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1200 IOs, 630016000 bytes 2023-10-11T10:57:23.833 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1300 IOs, 661042176 bytes 2023-10-11T10:57:24.031 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1400 IOs, 716187648 bytes 2023-10-11T10:57:24.247 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1500 IOs, 800538624 bytes 2023-10-11T10:57:24.396 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1600 IOs, 811467776 bytes 2023-10-11T10:57:24.523 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1700 IOs, 825284608 bytes 2023-10-11T10:57:24.679 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1800 IOs, 857612288 bytes 2023-10-11T10:57:24.930 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1900 IOs, 931814400 bytes 2023-10-11T10:57:25.191 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2000 IOs, 1010198528 bytes 2023-10-11T10:57:25.424 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2100 IOs, 1081515008 bytes 2023-10-11T10:57:25.556 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2200 IOs, 1098238976 bytes 2023-10-11T10:57:25.645 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2300 IOs, 1098848256 bytes 2023-10-11T10:57:25.868 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2400 IOs, 1164346368 bytes 2023-10-11T10:57:26.020 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2500 IOs, 1204656128 bytes 2023-10-11T10:57:26.236 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2600 IOs, 1260940288 bytes 2023-10-11T10:57:26.417 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2700 IOs, 1304670208 bytes 2023-10-11T10:57:26.662 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2800 IOs, 1381816320 bytes 2023-10-11T10:57:26.803 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2900 IOs, 1406738432 bytes 2023-10-11T10:57:26.901 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3000 IOs, 1407336448 bytes 2023-10-11T10:57:27.018 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3100 IOs, 1417949184 bytes 2023-10-11T10:57:27.195 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3200 IOs, 1457455104 bytes 2023-10-11T10:57:27.225 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ error_code=124 2023-10-11T10:57:27.226 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ set -e 2023-10-11T10:57:27.226 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' 124 -eq 124 ']' 2023-10-11T10:57:27.226 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T10:57:27.227 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T10:57:27.227 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_20 2023-10-11T10:57:27.227 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ create_image_and_enable_mirror cluster2 mirror image_20 snapshot 128M 2023-10-11T10:57:27.227 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:57:27.228 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:57:27.228 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:57:27.228 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:57:27.228 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_20 2023-10-11T10:57:27.229 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:57:27.229 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local mode=snapshot 2023-10-11T10:57:27.229 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n snapshot ']' 2023-10-11T10:57:27.229 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:57:27.229 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ create_image cluster2 mirror image_20 128M 2023-10-11T10:57:27.230 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:57:27.230 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:57:27.230 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:57:27.230 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:57:27.230 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_20 2023-10-11T10:57:27.231 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:57:27.231 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local size=128 2023-10-11T10:57:27.231 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n 128M ']' 2023-10-11T10:57:27.231 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ size=128M 2023-10-11T10:57:27.231 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:57:27.232 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_20 2023-10-11T10:57:27.278 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' image = image ']' 2023-10-11T10:57:27.278 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ enable_mirror cluster2 mirror image_20 snapshot 2023-10-11T10:57:27.278 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:57:27.278 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:57:27.279 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_20 2023-10-11T10:57:27.279 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local mode=snapshot 2023-10-11T10:57:27.279 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_20 snapshot 2023-10-11T10:57:27.441 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:Mirroring enabled 2023-10-11T10:57:27.450 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 info mirror/image_20 2023-10-11T10:57:27.485 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:rbd image 'image_20': 2023-10-11T10:57:27.485 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: size 128 MiB in 32 objects 2023-10-11T10:57:27.485 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: order 22 (4 MiB objects) 2023-10-11T10:57:27.486 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: snapshot_count: 1 2023-10-11T10:57:27.486 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: id: 12f21c8e4fcd 2023-10-11T10:57:27.486 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: block_name_prefix: rbd_data.12f21c8e4fcd 2023-10-11T10:57:27.486 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: format: 2 2023-10-11T10:57:27.486 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-10-11T10:57:27.487 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: op_features: 2023-10-11T10:57:27.487 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: flags: 2023-10-11T10:57:27.487 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: create_timestamp: Wed Oct 11 10:57:27 2023 2023-10-11T10:57:27.487 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: access_timestamp: Wed Oct 11 10:57:27 2023 2023-10-11T10:57:27.487 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: modify_timestamp: Wed Oct 11 10:57:27 2023 2023-10-11T10:57:27.487 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring state: enabled 2023-10-11T10:57:27.488 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring mode: snapshot 2023-10-11T10:57:27.488 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring global id: 46b9e65d-4379-44aa-a3c7-83f3b477d8e6 2023-10-11T10:57:27.488 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring primary: true 2023-10-11T10:57:27.489 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n '' ']' 2023-10-11T10:57:27.489 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ stress_write_image cluster2 mirror image_20 2023-10-11T10:57:27.489 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:57:27.490 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:57:27.490 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_20 2023-10-11T10:57:27.490 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-10-11T10:57:27.491 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local duration=8 2023-10-11T10:57:27.491 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ set +e 2023-10-11T10:57:27.491 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_20 --debug-rbd=20 --debug-journaler=20 2023-10-11T10:57:27.667 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:100 IOs, 612352 bytes 2023-10-11T10:57:27.875 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:200 IOs, 75029504 bytes 2023-10-11T10:57:28.056 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:300 IOs, 94164992 bytes 2023-10-11T10:57:28.264 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:400 IOs, 169520128 bytes 2023-10-11T10:57:28.459 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:500 IOs, 207523840 bytes 2023-10-11T10:57:28.642 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:600 IOs, 252249088 bytes 2023-10-11T10:57:28.798 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:700 IOs, 281419776 bytes 2023-10-11T10:57:29.001 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:800 IOs, 340903936 bytes 2023-10-11T10:57:29.179 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:900 IOs, 377096192 bytes 2023-10-11T10:57:29.328 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1000 IOs, 415063040 bytes 2023-10-11T10:57:29.476 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1100 IOs, 435092480 bytes 2023-10-11T10:57:29.610 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1200 IOs, 455151616 bytes 2023-10-11T10:57:29.807 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1300 IOs, 501490688 bytes 2023-10-11T10:57:30.109 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1400 IOs, 603348992 bytes 2023-10-11T10:57:30.272 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1500 IOs, 638981120 bytes 2023-10-11T10:57:30.454 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1600 IOs, 685465600 bytes 2023-10-11T10:57:30.684 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1700 IOs, 754556928 bytes 2023-10-11T10:57:30.828 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1800 IOs, 772925440 bytes 2023-10-11T10:57:30.939 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1900 IOs, 797700096 bytes 2023-10-11T10:57:31.169 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2000 IOs, 851691520 bytes 2023-10-11T10:57:31.363 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2100 IOs, 918277120 bytes 2023-10-11T10:57:31.638 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2200 IOs, 985798656 bytes 2023-10-11T10:57:31.826 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2300 IOs, 1036831744 bytes 2023-10-11T10:57:32.050 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2400 IOs, 1100342272 bytes 2023-10-11T10:57:32.323 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2500 IOs, 1185792000 bytes 2023-10-11T10:57:32.647 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2600 IOs, 1309083648 bytes 2023-10-11T10:57:32.783 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2700 IOs, 1321448448 bytes 2023-10-11T10:57:33.025 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2800 IOs, 1389843456 bytes 2023-10-11T10:57:33.205 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2900 IOs, 1424453632 bytes 2023-10-11T10:57:33.312 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3000 IOs, 1425065984 bytes 2023-10-11T10:57:33.493 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3100 IOs, 1467728896 bytes 2023-10-11T10:57:33.636 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3200 IOs, 1488966656 bytes 2023-10-11T10:57:33.840 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3300 IOs, 1541771264 bytes 2023-10-11T10:57:34.032 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3400 IOs, 1586318336 bytes 2023-10-11T10:57:34.188 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3500 IOs, 1618046976 bytes 2023-10-11T10:57:34.337 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3600 IOs, 1646073856 bytes 2023-10-11T10:57:34.601 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3700 IOs, 1736872960 bytes 2023-10-11T10:57:34.716 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3800 IOs, 1741858816 bytes 2023-10-11T10:57:34.908 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3900 IOs, 1792691200 bytes 2023-10-11T10:57:35.011 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4000 IOs, 1794350080 bytes 2023-10-11T10:57:35.108 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4100 IOs, 1794963456 bytes 2023-10-11T10:57:35.286 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4200 IOs, 1839037440 bytes 2023-10-11T10:57:35.489 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4300 IOs, 1901177856 bytes 2023-10-11T10:57:35.495 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ error_code=124 2023-10-11T10:57:35.496 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ set -e 2023-10-11T10:57:35.496 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' 124 -eq 124 ']' 2023-10-11T10:57:35.496 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T10:57:35.496 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T10:57:35.496 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_21 2023-10-11T10:57:35.497 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ create_image_and_enable_mirror cluster2 mirror image_21 snapshot 128M 2023-10-11T10:57:35.497 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:57:35.497 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:57:35.497 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:57:35.497 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:57:35.498 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_21 2023-10-11T10:57:35.498 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:57:35.498 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local mode=snapshot 2023-10-11T10:57:35.498 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n snapshot ']' 2023-10-11T10:57:35.498 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:57:35.498 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ create_image cluster2 mirror image_21 128M 2023-10-11T10:57:35.499 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:57:35.499 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:57:35.499 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:57:35.499 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:57:35.499 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_21 2023-10-11T10:57:35.500 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:57:35.500 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local size=128 2023-10-11T10:57:35.500 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n 128M ']' 2023-10-11T10:57:35.500 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ size=128M 2023-10-11T10:57:35.500 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:57:35.500 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_21 2023-10-11T10:57:35.544 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' image = image ']' 2023-10-11T10:57:35.544 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ enable_mirror cluster2 mirror image_21 snapshot 2023-10-11T10:57:35.544 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:57:35.544 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:57:35.545 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_21 2023-10-11T10:57:35.545 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local mode=snapshot 2023-10-11T10:57:35.545 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_21 snapshot 2023-10-11T10:57:36.363 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:Mirroring enabled 2023-10-11T10:57:36.373 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 info mirror/image_21 2023-10-11T10:57:36.408 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:rbd image 'image_21': 2023-10-11T10:57:36.409 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: size 128 MiB in 32 objects 2023-10-11T10:57:36.409 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: order 22 (4 MiB objects) 2023-10-11T10:57:36.409 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: snapshot_count: 1 2023-10-11T10:57:36.409 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: id: 12fe290b6c17 2023-10-11T10:57:36.409 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: block_name_prefix: rbd_data.12fe290b6c17 2023-10-11T10:57:36.410 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: format: 2 2023-10-11T10:57:36.410 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-10-11T10:57:36.410 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: op_features: 2023-10-11T10:57:36.410 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: flags: 2023-10-11T10:57:36.410 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: create_timestamp: Wed Oct 11 10:57:35 2023 2023-10-11T10:57:36.411 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: access_timestamp: Wed Oct 11 10:57:35 2023 2023-10-11T10:57:36.411 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: modify_timestamp: Wed Oct 11 10:57:35 2023 2023-10-11T10:57:36.411 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring state: enabled 2023-10-11T10:57:36.411 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring mode: snapshot 2023-10-11T10:57:36.411 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring global id: 11e7aef5-a626-4f47-9643-5ab9888eea8f 2023-10-11T10:57:36.411 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring primary: true 2023-10-11T10:57:36.412 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n '' ']' 2023-10-11T10:57:36.412 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ stress_write_image cluster2 mirror image_21 2023-10-11T10:57:36.413 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:57:36.413 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:57:36.413 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_21 2023-10-11T10:57:36.413 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-10-11T10:57:36.414 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local duration=10 2023-10-11T10:57:36.414 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ set +e 2023-10-11T10:57:36.414 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_21 --debug-rbd=20 --debug-journaler=20 2023-10-11T10:57:36.706 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:100 IOs, 67820544 bytes 2023-10-11T10:57:36.970 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:200 IOs, 157676544 bytes 2023-10-11T10:57:37.150 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:300 IOs, 211007488 bytes 2023-10-11T10:57:37.282 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:400 IOs, 214687744 bytes 2023-10-11T10:57:37.491 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:500 IOs, 266828800 bytes 2023-10-11T10:57:37.715 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:600 IOs, 331457536 bytes 2023-10-11T10:57:37.828 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:700 IOs, 343248896 bytes 2023-10-11T10:57:37.976 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:800 IOs, 368611328 bytes 2023-10-11T10:57:38.165 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:900 IOs, 422991872 bytes 2023-10-11T10:57:38.375 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1000 IOs, 480314368 bytes 2023-10-11T10:57:38.539 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1100 IOs, 522583040 bytes 2023-10-11T10:57:38.719 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1200 IOs, 551826432 bytes 2023-10-11T10:57:38.916 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1300 IOs, 603732992 bytes 2023-10-11T10:57:39.175 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1400 IOs, 688896000 bytes 2023-10-11T10:57:39.377 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1500 IOs, 743980032 bytes 2023-10-11T10:57:39.497 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1600 IOs, 754235392 bytes 2023-10-11T10:57:39.703 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1700 IOs, 801183744 bytes 2023-10-11T10:57:39.985 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1800 IOs, 897789952 bytes 2023-10-11T10:57:40.149 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1900 IOs, 926932992 bytes 2023-10-11T10:57:40.297 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2000 IOs, 946710528 bytes 2023-10-11T10:57:40.413 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2100 IOs, 957573120 bytes 2023-10-11T10:57:40.613 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2200 IOs, 996613120 bytes 2023-10-11T10:57:40.776 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2300 IOs, 1026688000 bytes 2023-10-11T10:57:40.917 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2400 IOs, 1047214080 bytes 2023-10-11T10:57:41.023 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2500 IOs, 1047799808 bytes 2023-10-11T10:57:41.177 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2600 IOs, 1068896256 bytes 2023-10-11T10:57:41.382 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2700 IOs, 1116306432 bytes 2023-10-11T10:57:41.543 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2800 IOs, 1142734848 bytes 2023-10-11T10:57:41.651 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2900 IOs, 1143375872 bytes 2023-10-11T10:57:41.757 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3000 IOs, 1144281088 bytes 2023-10-11T10:57:41.984 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3100 IOs, 1205989376 bytes 2023-10-11T10:57:42.217 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3200 IOs, 1264086016 bytes 2023-10-11T10:57:42.403 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3300 IOs, 1308018688 bytes 2023-10-11T10:57:42.510 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3400 IOs, 1314003968 bytes 2023-10-11T10:57:42.870 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3500 IOs, 1415716864 bytes 2023-10-11T10:57:43.031 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3600 IOs, 1439401984 bytes 2023-10-11T10:57:43.156 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3700 IOs, 1453207552 bytes 2023-10-11T10:57:43.374 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3800 IOs, 1509394432 bytes 2023-10-11T10:57:43.555 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3900 IOs, 1531154432 bytes 2023-10-11T10:57:43.864 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4000 IOs, 1621599232 bytes 2023-10-11T10:57:44.119 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4100 IOs, 1690259456 bytes 2023-10-11T10:57:44.273 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4200 IOs, 1738786816 bytes 2023-10-11T10:57:44.416 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4300 IOs, 1749938176 bytes 2023-10-11T10:57:44.592 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4400 IOs, 1783188480 bytes 2023-10-11T10:57:44.761 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4500 IOs, 1815584768 bytes 2023-10-11T10:57:45.040 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4600 IOs, 1898212352 bytes 2023-10-11T10:57:45.215 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4700 IOs, 1937939456 bytes 2023-10-11T10:57:45.385 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4800 IOs, 1963389952 bytes 2023-10-11T10:57:45.535 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4900 IOs, 1988264960 bytes 2023-10-11T10:57:45.718 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5000 IOs, 2022952960 bytes 2023-10-11T10:57:45.913 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5100 IOs, 2083343360 bytes 2023-10-11T10:57:46.286 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5200 IOs, 2122839040 bytes 2023-10-11T10:57:46.419 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ error_code=124 2023-10-11T10:57:46.419 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ set -e 2023-10-11T10:57:46.419 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' 124 -eq 124 ']' 2023-10-11T10:57:46.420 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T10:57:46.420 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T10:57:46.420 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_22 2023-10-11T10:57:46.420 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ create_image_and_enable_mirror cluster2 mirror image_22 snapshot 128M 2023-10-11T10:57:46.420 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:57:46.421 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:57:46.421 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:57:46.421 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:57:46.421 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_22 2023-10-11T10:57:46.421 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:57:46.421 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local mode=snapshot 2023-10-11T10:57:46.422 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n snapshot ']' 2023-10-11T10:57:46.422 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:57:46.422 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ create_image cluster2 mirror image_22 128M 2023-10-11T10:57:46.422 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:57:46.422 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:57:46.422 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:57:46.423 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:57:46.423 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_22 2023-10-11T10:57:46.423 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:57:46.423 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local size=128 2023-10-11T10:57:46.423 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n 128M ']' 2023-10-11T10:57:46.424 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ size=128M 2023-10-11T10:57:46.424 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:57:46.424 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_22 2023-10-11T10:57:46.467 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' image = image ']' 2023-10-11T10:57:46.467 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ enable_mirror cluster2 mirror image_22 snapshot 2023-10-11T10:57:46.467 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:57:46.467 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:57:46.467 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_22 2023-10-11T10:57:46.468 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local mode=snapshot 2023-10-11T10:57:46.468 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_22 snapshot 2023-10-11T10:57:47.407 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:Mirroring enabled 2023-10-11T10:57:47.417 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 info mirror/image_22 2023-10-11T10:57:47.454 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:rbd image 'image_22': 2023-10-11T10:57:47.454 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: size 128 MiB in 32 objects 2023-10-11T10:57:47.455 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: order 22 (4 MiB objects) 2023-10-11T10:57:47.455 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: snapshot_count: 1 2023-10-11T10:57:47.455 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: id: 130a960cd1a2 2023-10-11T10:57:47.455 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: block_name_prefix: rbd_data.130a960cd1a2 2023-10-11T10:57:47.455 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: format: 2 2023-10-11T10:57:47.456 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-10-11T10:57:47.456 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: op_features: 2023-10-11T10:57:47.456 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: flags: 2023-10-11T10:57:47.456 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: create_timestamp: Wed Oct 11 10:57:46 2023 2023-10-11T10:57:47.456 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: access_timestamp: Wed Oct 11 10:57:46 2023 2023-10-11T10:57:47.457 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: modify_timestamp: Wed Oct 11 10:57:46 2023 2023-10-11T10:57:47.457 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring state: enabled 2023-10-11T10:57:47.457 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring mode: snapshot 2023-10-11T10:57:47.457 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring global id: 1288fb83-4aad-4242-bdd2-1944a6794332 2023-10-11T10:57:47.457 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring primary: true 2023-10-11T10:57:47.458 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n '' ']' 2023-10-11T10:57:47.458 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ stress_write_image cluster2 mirror image_22 2023-10-11T10:57:47.458 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:57:47.459 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:57:47.459 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_22 2023-10-11T10:57:47.459 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-10-11T10:57:47.460 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local duration=12 2023-10-11T10:57:47.460 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ set +e 2023-10-11T10:57:47.460 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_22 --debug-rbd=20 --debug-journaler=20 2023-10-11T10:57:47.782 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:100 IOs, 74135552 bytes 2023-10-11T10:57:47.936 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:200 IOs, 119150592 bytes 2023-10-11T10:57:48.400 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:300 IOs, 169668608 bytes 2023-10-11T10:57:48.523 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:400 IOs, 204664832 bytes 2023-10-11T10:57:48.723 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:500 IOs, 241275904 bytes 2023-10-11T10:57:48.992 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:600 IOs, 336816128 bytes 2023-10-11T10:57:49.109 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:700 IOs, 348773376 bytes 2023-10-11T10:57:49.229 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:800 IOs, 356776960 bytes 2023-10-11T10:57:49.420 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:900 IOs, 405618688 bytes 2023-10-11T10:57:49.662 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1000 IOs, 487877632 bytes 2023-10-11T10:57:49.841 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1100 IOs, 525077504 bytes 2023-10-11T10:57:50.013 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1200 IOs, 568325120 bytes 2023-10-11T10:57:50.193 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1300 IOs, 612270080 bytes 2023-10-11T10:57:50.432 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1400 IOs, 683298816 bytes 2023-10-11T10:57:50.663 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1500 IOs, 757122048 bytes 2023-10-11T10:57:50.783 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1600 IOs, 767524864 bytes 2023-10-11T10:57:50.914 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1700 IOs, 782727168 bytes 2023-10-11T10:57:51.167 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1800 IOs, 848369664 bytes 2023-10-11T10:57:51.352 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1900 IOs, 897960960 bytes 2023-10-11T10:57:51.546 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2000 IOs, 942792704 bytes 2023-10-11T10:57:51.717 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2100 IOs, 982005760 bytes 2023-10-11T10:57:51.979 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2200 IOs, 1061067776 bytes 2023-10-11T10:57:52.109 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2300 IOs, 1080429568 bytes 2023-10-11T10:57:52.509 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2400 IOs, 1139830784 bytes 2023-10-11T10:57:52.656 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2500 IOs, 1162754048 bytes 2023-10-11T10:57:52.857 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2600 IOs, 1219936256 bytes 2023-10-11T10:57:52.971 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2700 IOs, 1222308864 bytes 2023-10-11T10:57:53.081 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2800 IOs, 1235337216 bytes 2023-10-11T10:57:53.238 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2900 IOs, 1258712064 bytes 2023-10-11T10:57:53.406 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3000 IOs, 1298594816 bytes 2023-10-11T10:57:53.801 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3100 IOs, 1346923520 bytes 2023-10-11T10:57:53.902 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3200 IOs, 1347829760 bytes 2023-10-11T10:57:54.015 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3300 IOs, 1358746624 bytes 2023-10-11T10:57:54.191 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3400 IOs, 1398123520 bytes 2023-10-11T10:57:54.308 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3500 IOs, 1400566784 bytes 2023-10-11T10:57:54.560 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3600 IOs, 1486157824 bytes 2023-10-11T10:57:54.790 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3700 IOs, 1551499264 bytes 2023-10-11T10:57:55.011 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3800 IOs, 1616668672 bytes 2023-10-11T10:57:55.260 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3900 IOs, 1670363136 bytes 2023-10-11T10:57:55.417 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4000 IOs, 1696384000 bytes 2023-10-11T10:57:55.602 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4100 IOs, 1759876096 bytes 2023-10-11T10:57:55.806 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4200 IOs, 1801904128 bytes 2023-10-11T10:57:55.945 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4300 IOs, 1827284992 bytes 2023-10-11T10:57:56.069 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4400 IOs, 1840038912 bytes 2023-10-11T10:57:56.218 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4500 IOs, 1869191168 bytes 2023-10-11T10:57:56.438 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4600 IOs, 1937875968 bytes 2023-10-11T10:57:56.640 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4700 IOs, 1992355840 bytes 2023-10-11T10:57:56.779 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4800 IOs, 2009719808 bytes 2023-10-11T10:57:56.990 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4900 IOs, 2066142208 bytes 2023-10-11T10:57:57.157 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5000 IOs, 2118729728 bytes 2023-10-11T10:57:57.376 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5100 IOs, 2182822912 bytes 2023-10-11T10:57:57.543 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5200 IOs, 2193831936 bytes 2023-10-11T10:57:57.743 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5300 IOs, 2247192576 bytes 2023-10-11T10:57:57.930 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5400 IOs, 2286345216 bytes 2023-10-11T10:57:58.093 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5500 IOs, 2321260544 bytes 2023-10-11T10:57:58.226 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5600 IOs, 2336874496 bytes 2023-10-11T10:57:58.363 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5700 IOs, 2356753408 bytes 2023-10-11T10:57:58.626 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5800 IOs, 2431392768 bytes 2023-10-11T10:57:58.751 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5900 IOs, 2440912896 bytes 2023-10-11T10:57:58.922 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6000 IOs, 2474987520 bytes 2023-10-11T10:57:59.102 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6100 IOs, 2512686080 bytes 2023-10-11T10:57:59.259 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6200 IOs, 2548988928 bytes 2023-10-11T10:57:59.467 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ error_code=124 2023-10-11T10:57:59.468 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ set -e 2023-10-11T10:57:59.468 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' 124 -eq 124 ']' 2023-10-11T10:57:59.468 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T10:57:59.468 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T10:57:59.468 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_23 2023-10-11T10:57:59.469 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ create_image_and_enable_mirror cluster2 mirror image_23 snapshot 128M 2023-10-11T10:57:59.469 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:57:59.469 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:57:59.469 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:57:59.469 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:57:59.469 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_23 2023-10-11T10:57:59.470 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:57:59.470 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local mode=snapshot 2023-10-11T10:57:59.470 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n snapshot ']' 2023-10-11T10:57:59.470 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:57:59.470 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ create_image cluster2 mirror image_23 128M 2023-10-11T10:57:59.471 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:57:59.471 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:57:59.471 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:57:59.471 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:57:59.471 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_23 2023-10-11T10:57:59.472 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:57:59.472 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local size=128 2023-10-11T10:57:59.472 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n 128M ']' 2023-10-11T10:57:59.472 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ size=128M 2023-10-11T10:57:59.472 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:57:59.472 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_23 2023-10-11T10:57:59.520 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' image = image ']' 2023-10-11T10:57:59.520 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ enable_mirror cluster2 mirror image_23 snapshot 2023-10-11T10:57:59.520 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:57:59.521 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:57:59.521 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_23 2023-10-11T10:57:59.521 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local mode=snapshot 2023-10-11T10:57:59.521 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_23 snapshot 2023-10-11T10:58:00.461 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:Mirroring enabled 2023-10-11T10:58:00.472 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 info mirror/image_23 2023-10-11T10:58:00.509 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:rbd image 'image_23': 2023-10-11T10:58:00.509 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: size 128 MiB in 32 objects 2023-10-11T10:58:00.510 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: order 22 (4 MiB objects) 2023-10-11T10:58:00.510 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: snapshot_count: 1 2023-10-11T10:58:00.510 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: id: 1316b88a16e5 2023-10-11T10:58:00.510 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: block_name_prefix: rbd_data.1316b88a16e5 2023-10-11T10:58:00.511 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: format: 2 2023-10-11T10:58:00.511 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-10-11T10:58:00.511 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: op_features: 2023-10-11T10:58:00.511 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: flags: 2023-10-11T10:58:00.511 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: create_timestamp: Wed Oct 11 10:57:59 2023 2023-10-11T10:58:00.511 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: access_timestamp: Wed Oct 11 10:57:59 2023 2023-10-11T10:58:00.512 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: modify_timestamp: Wed Oct 11 10:57:59 2023 2023-10-11T10:58:00.512 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring state: enabled 2023-10-11T10:58:00.512 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring mode: snapshot 2023-10-11T10:58:00.512 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring global id: b3738f1e-cf2d-40b8-80a2-69299f4bee7b 2023-10-11T10:58:00.512 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring primary: true 2023-10-11T10:58:00.513 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n '' ']' 2023-10-11T10:58:00.514 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ stress_write_image cluster2 mirror image_23 2023-10-11T10:58:00.514 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:58:00.514 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:58:00.514 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_23 2023-10-11T10:58:00.514 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-10-11T10:58:00.515 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local duration=5 2023-10-11T10:58:00.515 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ set +e 2023-10-11T10:58:00.515 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_23 --debug-rbd=20 --debug-journaler=20 2023-10-11T10:58:00.756 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:100 IOs, 56685568 bytes 2023-10-11T10:58:00.957 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:200 IOs, 89097216 bytes 2023-10-11T10:58:01.131 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:300 IOs, 130061312 bytes 2023-10-11T10:58:01.329 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:400 IOs, 190105600 bytes 2023-10-11T10:58:01.451 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:500 IOs, 210629632 bytes 2023-10-11T10:58:01.643 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:600 IOs, 258229248 bytes 2023-10-11T10:58:01.804 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:700 IOs, 283277312 bytes 2023-10-11T10:58:01.934 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:800 IOs, 300584960 bytes 2023-10-11T10:58:02.096 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:900 IOs, 336262144 bytes 2023-10-11T10:58:02.257 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1000 IOs, 370805760 bytes 2023-10-11T10:58:02.442 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1100 IOs, 421829632 bytes 2023-10-11T10:58:02.667 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1200 IOs, 476219392 bytes 2023-10-11T10:58:02.893 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1300 IOs, 543652864 bytes 2023-10-11T10:58:03.142 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1400 IOs, 610566144 bytes 2023-10-11T10:58:03.363 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1500 IOs, 674503680 bytes 2023-10-11T10:58:03.533 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1600 IOs, 715772928 bytes 2023-10-11T10:58:03.546 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2023-10-11T10:58:03.546 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2023-10-11T10:58:03.547 INFO:tasks.rbd_mirror_thrash:waiting for 8 secs before reviving daemons 2023-10-11T10:58:03.547 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi111.stderr:2023-10-11T10:58:03.546+0000 7f8961993700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 113606) UID: 1000 2023-10-11T10:58:03.670 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1700 IOs, 754142208 bytes 2023-10-11T10:58:03.715 INFO:tasks.ceph.cluster2.osd.0.smithi111.stderr:2023-10-11T10:58:03.712+0000 7f80be969700 -1 reset not still connected to 0x5555c54b8000 2023-10-11T10:58:03.716 INFO:tasks.ceph.cluster2.osd.0.smithi111.stderr:2023-10-11T10:58:03.712+0000 7f80be969700 -1 reset not still connected to 0x5555c715b040 2023-10-11T10:58:03.850 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1800 IOs, 765891584 bytes 2023-10-11T10:58:03.985 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1900 IOs, 789368832 bytes 2023-10-11T10:58:04.176 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2000 IOs, 831010816 bytes 2023-10-11T10:58:04.347 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2100 IOs, 869493760 bytes 2023-10-11T10:58:04.494 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2200 IOs, 896868352 bytes 2023-10-11T10:58:04.726 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2300 IOs, 955104256 bytes 2023-10-11T10:58:04.840 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2400 IOs, 963733504 bytes 2023-10-11T10:58:04.953 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2500 IOs, 969577472 bytes 2023-10-11T10:58:05.190 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2600 IOs, 1041992704 bytes 2023-10-11T10:58:05.342 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2700 IOs, 1070182400 bytes 2023-10-11T10:58:05.518 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ error_code=124 2023-10-11T10:58:05.518 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ set -e 2023-10-11T10:58:05.519 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' 124 -eq 124 ']' 2023-10-11T10:58:05.519 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T10:58:05.519 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T10:58:05.519 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_24 2023-10-11T10:58:05.519 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ create_image_and_enable_mirror cluster2 mirror image_24 snapshot 128M 2023-10-11T10:58:05.520 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:58:05.520 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:58:05.520 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:58:05.520 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:58:05.520 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_24 2023-10-11T10:58:05.521 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:58:05.521 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local mode=snapshot 2023-10-11T10:58:05.521 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n snapshot ']' 2023-10-11T10:58:05.521 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:58:05.521 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ create_image cluster2 mirror image_24 128M 2023-10-11T10:58:05.521 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:58:05.522 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:58:05.522 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:58:05.522 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:58:05.522 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_24 2023-10-11T10:58:05.523 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:58:05.523 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local size=128 2023-10-11T10:58:05.523 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n 128M ']' 2023-10-11T10:58:05.523 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ size=128M 2023-10-11T10:58:05.523 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:58:05.523 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_24 2023-10-11T10:58:05.565 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' image = image ']' 2023-10-11T10:58:05.565 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ enable_mirror cluster2 mirror image_24 snapshot 2023-10-11T10:58:05.565 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:58:05.566 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:58:05.566 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_24 2023-10-11T10:58:05.566 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local mode=snapshot 2023-10-11T10:58:05.566 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_24 snapshot 2023-10-11T10:58:06.496 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:Mirroring enabled 2023-10-11T10:58:06.507 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 info mirror/image_24 2023-10-11T10:58:06.546 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:rbd image 'image_24': 2023-10-11T10:58:06.546 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: size 128 MiB in 32 objects 2023-10-11T10:58:06.546 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: order 22 (4 MiB objects) 2023-10-11T10:58:06.546 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: snapshot_count: 1 2023-10-11T10:58:06.547 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: id: 1322488f3b42 2023-10-11T10:58:06.547 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: block_name_prefix: rbd_data.1322488f3b42 2023-10-11T10:58:06.547 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: format: 2 2023-10-11T10:58:06.547 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-10-11T10:58:06.547 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: op_features: 2023-10-11T10:58:06.547 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: flags: 2023-10-11T10:58:06.548 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: create_timestamp: Wed Oct 11 10:58:05 2023 2023-10-11T10:58:06.548 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: access_timestamp: Wed Oct 11 10:58:05 2023 2023-10-11T10:58:06.548 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: modify_timestamp: Wed Oct 11 10:58:05 2023 2023-10-11T10:58:06.548 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring state: enabled 2023-10-11T10:58:06.548 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring mode: snapshot 2023-10-11T10:58:06.549 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring global id: 1d287991-2520-4320-9427-8214deba681d 2023-10-11T10:58:06.549 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring primary: true 2023-10-11T10:58:06.550 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n '' ']' 2023-10-11T10:58:06.550 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ stress_write_image cluster2 mirror image_24 2023-10-11T10:58:06.550 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:58:06.550 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:58:06.550 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_24 2023-10-11T10:58:06.551 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-10-11T10:58:06.551 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local duration=7 2023-10-11T10:58:06.551 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ set +e 2023-10-11T10:58:06.552 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_24 --debug-rbd=20 --debug-journaler=20 2023-10-11T10:58:06.857 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:100 IOs, 73626624 bytes 2023-10-11T10:58:07.028 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:200 IOs, 109364224 bytes 2023-10-11T10:58:07.196 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:300 IOs, 148408320 bytes 2023-10-11T10:58:07.416 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:400 IOs, 216167424 bytes 2023-10-11T10:58:07.572 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:500 IOs, 248555520 bytes 2023-10-11T10:58:07.729 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:600 IOs, 281594880 bytes 2023-10-11T10:58:08.021 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:700 IOs, 353192960 bytes 2023-10-11T10:58:08.123 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:800 IOs, 353820672 bytes 2023-10-11T10:58:08.268 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:900 IOs, 375685120 bytes 2023-10-11T10:58:08.424 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1000 IOs, 392947712 bytes 2023-10-11T10:58:08.621 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1100 IOs, 444498944 bytes 2023-10-11T10:58:08.785 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1200 IOs, 475902976 bytes 2023-10-11T10:58:08.924 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1300 IOs, 497357824 bytes 2023-10-11T10:58:09.036 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1400 IOs, 501309440 bytes 2023-10-11T10:58:09.190 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1500 IOs, 528721920 bytes 2023-10-11T10:58:09.443 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1600 IOs, 597112832 bytes 2023-10-11T10:58:09.604 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1700 IOs, 629668864 bytes 2023-10-11T10:58:09.795 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1800 IOs, 678794240 bytes 2023-10-11T10:58:10.007 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1900 IOs, 736059392 bytes 2023-10-11T10:58:10.206 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2000 IOs, 796995584 bytes 2023-10-11T10:58:10.359 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2100 IOs, 809891840 bytes 2023-10-11T10:58:10.523 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2200 IOs, 836185088 bytes 2023-10-11T10:58:10.746 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2300 IOs, 915351552 bytes 2023-10-11T10:58:10.948 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2400 IOs, 950558720 bytes 2023-10-11T10:58:11.119 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2500 IOs, 986553344 bytes 2023-10-11T10:58:11.311 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2600 IOs, 1033039872 bytes 2023-10-11T10:58:11.547 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2023-10-11T10:58:11.547 INFO:teuthology.orchestra.run:waiting for 600 2023-10-11T10:58:11.548 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2023-10-11T10:58:11.548 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2023-10-11T10:58:11.548 DEBUG:teuthology.orchestra.run.smithi111:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2023-10-11T10:58:11.556 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2023-10-11T10:58:11.556 INFO:tasks.rbd_mirror_thrash:waiting for 64 secs before thrashing 2023-10-11T10:58:11.604 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2700 IOs, 1138587648 bytes 2023-10-11T10:58:11.822 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2800 IOs, 1198291968 bytes 2023-10-11T10:58:11.946 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2900 IOs, 1203720192 bytes 2023-10-11T10:58:12.147 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3000 IOs, 1247072256 bytes 2023-10-11T10:58:12.418 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3100 IOs, 1319313408 bytes 2023-10-11T10:58:12.678 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3200 IOs, 1389956096 bytes 2023-10-11T10:58:12.882 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3300 IOs, 1455412224 bytes 2023-10-11T10:58:13.174 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3400 IOs, 1489697792 bytes 2023-10-11T10:58:13.400 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3500 IOs, 1546585088 bytes 2023-10-11T10:58:13.558 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ error_code=124 2023-10-11T10:58:13.558 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ set -e 2023-10-11T10:58:13.558 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' 124 -eq 124 ']' 2023-10-11T10:58:13.558 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T10:58:13.559 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T10:58:13.559 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_25 2023-10-11T10:58:13.559 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ create_image_and_enable_mirror cluster2 mirror image_25 snapshot 128M 2023-10-11T10:58:13.560 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:58:13.560 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:58:13.560 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:58:13.561 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:58:13.561 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_25 2023-10-11T10:58:13.561 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:58:13.561 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local mode=snapshot 2023-10-11T10:58:13.562 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n snapshot ']' 2023-10-11T10:58:13.562 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:58:13.562 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ create_image cluster2 mirror image_25 128M 2023-10-11T10:58:13.563 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:58:13.563 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:58:13.563 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:58:13.563 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:58:13.564 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_25 2023-10-11T10:58:13.565 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:58:13.566 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local size=128 2023-10-11T10:58:13.566 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n 128M ']' 2023-10-11T10:58:13.566 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ size=128M 2023-10-11T10:58:13.566 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:58:13.567 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_25 2023-10-11T10:58:13.603 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' image = image ']' 2023-10-11T10:58:13.604 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ enable_mirror cluster2 mirror image_25 snapshot 2023-10-11T10:58:13.604 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:58:13.604 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:58:13.604 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_25 2023-10-11T10:58:13.605 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local mode=snapshot 2023-10-11T10:58:13.605 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_25 snapshot 2023-10-11T10:58:14.395 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:Mirroring enabled 2023-10-11T10:58:14.405 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 info mirror/image_25 2023-10-11T10:58:14.443 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:rbd image 'image_25': 2023-10-11T10:58:14.443 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: size 128 MiB in 32 objects 2023-10-11T10:58:14.443 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: order 22 (4 MiB objects) 2023-10-11T10:58:14.443 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: snapshot_count: 1 2023-10-11T10:58:14.444 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: id: 1332cc39f6d9 2023-10-11T10:58:14.444 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: block_name_prefix: rbd_data.1332cc39f6d9 2023-10-11T10:58:14.444 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: format: 2 2023-10-11T10:58:14.444 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-10-11T10:58:14.444 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: op_features: 2023-10-11T10:58:14.445 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: flags: 2023-10-11T10:58:14.445 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: create_timestamp: Wed Oct 11 10:58:13 2023 2023-10-11T10:58:14.445 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: access_timestamp: Wed Oct 11 10:58:13 2023 2023-10-11T10:58:14.445 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: modify_timestamp: Wed Oct 11 10:58:13 2023 2023-10-11T10:58:14.445 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring state: enabled 2023-10-11T10:58:14.446 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring mode: snapshot 2023-10-11T10:58:14.446 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring global id: 48104021-9df0-46cf-a4dc-ab4fb67e4fe1 2023-10-11T10:58:14.446 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring primary: true 2023-10-11T10:58:14.447 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n '' ']' 2023-10-11T10:58:14.447 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ stress_write_image cluster2 mirror image_25 2023-10-11T10:58:14.447 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:58:14.448 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:58:14.448 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_25 2023-10-11T10:58:14.448 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-10-11T10:58:14.449 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local duration=9 2023-10-11T10:58:14.449 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ set +e 2023-10-11T10:58:14.449 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_25 --debug-rbd=20 --debug-journaler=20 2023-10-11T10:58:14.656 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:100 IOs, 20759552 bytes 2023-10-11T10:58:14.781 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:200 IOs, 36199424 bytes 2023-10-11T10:58:14.936 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:300 IOs, 66150400 bytes 2023-10-11T10:58:15.060 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:400 IOs, 80784384 bytes 2023-10-11T10:58:15.228 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:500 IOs, 119870464 bytes 2023-10-11T10:58:15.428 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:600 IOs, 175747072 bytes 2023-10-11T10:58:15.615 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:700 IOs, 223897600 bytes 2023-10-11T10:58:15.755 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:800 IOs, 243142656 bytes 2023-10-11T10:58:15.903 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:900 IOs, 269031424 bytes 2023-10-11T10:58:16.048 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1000 IOs, 291463168 bytes 2023-10-11T10:58:16.227 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1100 IOs, 322281472 bytes 2023-10-11T10:58:16.366 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1200 IOs, 358441984 bytes 2023-10-11T10:58:16.551 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1300 IOs, 376388608 bytes 2023-10-11T10:58:16.730 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1400 IOs, 415109120 bytes 2023-10-11T10:58:17.006 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1500 IOs, 509685760 bytes 2023-10-11T10:58:17.236 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1600 IOs, 558465024 bytes 2023-10-11T10:58:17.393 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1700 IOs, 585475072 bytes 2023-10-11T10:58:17.505 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1800 IOs, 593634304 bytes 2023-10-11T10:58:17.731 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1900 IOs, 659543040 bytes 2023-10-11T10:58:17.918 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2000 IOs, 700950528 bytes 2023-10-11T10:58:18.144 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2100 IOs, 765834240 bytes 2023-10-11T10:58:18.330 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2200 IOs, 800390144 bytes 2023-10-11T10:58:18.508 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2300 IOs, 854202368 bytes 2023-10-11T10:58:18.737 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2400 IOs, 909998080 bytes 2023-10-11T10:58:18.892 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2500 IOs, 926939136 bytes 2023-10-11T10:58:19.050 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2600 IOs, 960200704 bytes 2023-10-11T10:58:19.273 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2700 IOs, 1013462016 bytes 2023-10-11T10:58:19.424 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2800 IOs, 1034064896 bytes 2023-10-11T10:58:19.599 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2900 IOs, 1074442240 bytes 2023-10-11T10:58:19.696 INFO:tasks.ceph.cluster2.osd.1.smithi111.stderr:2023-10-11T10:58:19.695+0000 7f9628714700 -1 reset not still connected to 0x564240b47ad0 2023-10-11T10:58:19.696 INFO:tasks.ceph.cluster2.osd.1.smithi111.stderr:2023-10-11T10:58:19.695+0000 7f9628714700 -1 reset not still connected to 0x56424331aa90 2023-10-11T10:58:19.749 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3000 IOs, 1091535872 bytes 2023-10-11T10:58:19.880 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3100 IOs, 1107630080 bytes 2023-10-11T10:58:20.065 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3200 IOs, 1148101632 bytes 2023-10-11T10:58:20.236 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3300 IOs, 1193866240 bytes 2023-10-11T10:58:20.445 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3400 IOs, 1251241984 bytes 2023-10-11T10:58:20.638 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3500 IOs, 1303380992 bytes 2023-10-11T10:58:20.938 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3600 IOs, 1405521920 bytes 2023-10-11T10:58:21.058 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3700 IOs, 1422583808 bytes 2023-10-11T10:58:21.216 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3800 IOs, 1453499392 bytes 2023-10-11T10:58:21.416 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3900 IOs, 1502644224 bytes 2023-10-11T10:58:21.582 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4000 IOs, 1530841088 bytes 2023-10-11T10:58:21.774 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4100 IOs, 1580379136 bytes 2023-10-11T10:58:22.011 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4200 IOs, 1653579776 bytes 2023-10-11T10:58:22.197 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4300 IOs, 1693625344 bytes 2023-10-11T10:58:22.364 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4400 IOs, 1726853120 bytes 2023-10-11T10:58:22.601 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4500 IOs, 1792750592 bytes 2023-10-11T10:58:22.807 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4600 IOs, 1853250560 bytes 2023-10-11T10:58:22.990 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4700 IOs, 1899619328 bytes 2023-10-11T10:58:23.178 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4800 IOs, 1939907584 bytes 2023-10-11T10:58:23.436 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4900 IOs, 2016947200 bytes 2023-10-11T10:58:23.457 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ error_code=124 2023-10-11T10:58:23.458 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ set -e 2023-10-11T10:58:23.458 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' 124 -eq 124 ']' 2023-10-11T10:58:23.458 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T10:58:23.459 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T10:58:23.459 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_26 2023-10-11T10:58:23.459 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ create_image_and_enable_mirror cluster2 mirror image_26 snapshot 128M 2023-10-11T10:58:23.459 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:58:23.460 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:58:23.460 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:58:23.460 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:58:23.460 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_26 2023-10-11T10:58:23.461 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:58:23.461 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local mode=snapshot 2023-10-11T10:58:23.461 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n snapshot ']' 2023-10-11T10:58:23.461 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:58:23.462 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ create_image cluster2 mirror image_26 128M 2023-10-11T10:58:23.462 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:58:23.462 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:58:23.462 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:58:23.463 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:58:23.463 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_26 2023-10-11T10:58:23.463 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:58:23.464 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local size=128 2023-10-11T10:58:23.464 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n 128M ']' 2023-10-11T10:58:23.464 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ size=128M 2023-10-11T10:58:23.464 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:58:23.465 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_26 2023-10-11T10:58:23.507 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' image = image ']' 2023-10-11T10:58:23.508 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ enable_mirror cluster2 mirror image_26 snapshot 2023-10-11T10:58:23.508 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:58:23.508 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:58:23.509 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_26 2023-10-11T10:58:23.509 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local mode=snapshot 2023-10-11T10:58:23.509 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_26 snapshot 2023-10-11T10:58:24.456 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:Mirroring enabled 2023-10-11T10:58:24.466 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 info mirror/image_26 2023-10-11T10:58:24.501 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:rbd image 'image_26': 2023-10-11T10:58:24.501 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: size 128 MiB in 32 objects 2023-10-11T10:58:24.502 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: order 22 (4 MiB objects) 2023-10-11T10:58:24.502 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: snapshot_count: 1 2023-10-11T10:58:24.502 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: id: 133cf2e5f011 2023-10-11T10:58:24.502 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: block_name_prefix: rbd_data.133cf2e5f011 2023-10-11T10:58:24.502 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: format: 2 2023-10-11T10:58:24.503 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-10-11T10:58:24.503 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: op_features: 2023-10-11T10:58:24.503 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: flags: 2023-10-11T10:58:24.503 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: create_timestamp: Wed Oct 11 10:58:23 2023 2023-10-11T10:58:24.503 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: access_timestamp: Wed Oct 11 10:58:23 2023 2023-10-11T10:58:24.503 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: modify_timestamp: Wed Oct 11 10:58:23 2023 2023-10-11T10:58:24.504 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring state: enabled 2023-10-11T10:58:24.504 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring mode: snapshot 2023-10-11T10:58:24.504 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring global id: 76bafcc4-4e59-4baf-875a-22631193e31e 2023-10-11T10:58:24.504 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring primary: true 2023-10-11T10:58:24.505 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n '' ']' 2023-10-11T10:58:24.505 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ stress_write_image cluster2 mirror image_26 2023-10-11T10:58:24.505 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:58:24.506 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:58:24.506 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_26 2023-10-11T10:58:24.506 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-10-11T10:58:24.507 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local duration=12 2023-10-11T10:58:24.507 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ set +e 2023-10-11T10:58:24.507 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_26 --debug-rbd=20 --debug-journaler=20 2023-10-11T10:58:24.763 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:100 IOs, 44596224 bytes 2023-10-11T10:58:24.905 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:200 IOs, 66955264 bytes 2023-10-11T10:58:25.162 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:300 IOs, 149608448 bytes 2023-10-11T10:58:25.407 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:400 IOs, 221054976 bytes 2023-10-11T10:58:25.588 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:500 IOs, 261359616 bytes 2023-10-11T10:58:25.830 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:600 IOs, 331522048 bytes 2023-10-11T10:58:26.018 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:700 IOs, 375449600 bytes 2023-10-11T10:58:26.181 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:800 IOs, 402879488 bytes 2023-10-11T10:58:26.356 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:900 IOs, 445431808 bytes 2023-10-11T10:58:26.540 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1000 IOs, 484261888 bytes 2023-10-11T10:58:26.677 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1100 IOs, 501466112 bytes 2023-10-11T10:58:26.859 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1200 IOs, 544848896 bytes 2023-10-11T10:58:26.992 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1300 IOs, 565270528 bytes 2023-10-11T10:58:27.177 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1400 IOs, 602446848 bytes 2023-10-11T10:58:27.332 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1500 IOs, 627173376 bytes 2023-10-11T10:58:27.446 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1600 IOs, 627791872 bytes 2023-10-11T10:58:27.571 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1700 IOs, 636903424 bytes 2023-10-11T10:58:27.705 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1800 IOs, 650084352 bytes 2023-10-11T10:58:27.843 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1900 IOs, 679297024 bytes 2023-10-11T10:58:28.088 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2000 IOs, 746662912 bytes 2023-10-11T10:58:28.280 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2100 IOs, 794956800 bytes 2023-10-11T10:58:28.460 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2200 IOs, 846426112 bytes 2023-10-11T10:58:28.665 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2300 IOs, 885979136 bytes 2023-10-11T10:58:28.819 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2400 IOs, 917980160 bytes 2023-10-11T10:58:28.975 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2500 IOs, 943164416 bytes 2023-10-11T10:58:29.161 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2600 IOs, 987388928 bytes 2023-10-11T10:58:29.339 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2700 IOs, 1022219264 bytes 2023-10-11T10:58:29.533 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2800 IOs, 1080259584 bytes 2023-10-11T10:58:29.737 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2900 IOs, 1127422976 bytes 2023-10-11T10:58:29.910 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3000 IOs, 1163089920 bytes 2023-10-11T10:58:30.055 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3100 IOs, 1183267840 bytes 2023-10-11T10:58:30.251 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3200 IOs, 1234114560 bytes 2023-10-11T10:58:30.357 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3300 IOs, 1234724864 bytes 2023-10-11T10:58:30.526 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3400 IOs, 1293735936 bytes 2023-10-11T10:58:30.781 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3500 IOs, 1351163904 bytes 2023-10-11T10:58:30.945 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3600 IOs, 1374772224 bytes 2023-10-11T10:58:31.104 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3700 IOs, 1399825408 bytes 2023-10-11T10:58:31.253 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3800 IOs, 1426163712 bytes 2023-10-11T10:58:31.473 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3900 IOs, 1483300864 bytes 2023-10-11T10:58:31.597 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4000 IOs, 1516183552 bytes 2023-10-11T10:58:31.899 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4100 IOs, 1592171520 bytes 2023-10-11T10:58:32.156 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4200 IOs, 1666940928 bytes 2023-10-11T10:58:32.343 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4300 IOs, 1708961792 bytes 2023-10-11T10:58:32.586 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4400 IOs, 1776243712 bytes 2023-10-11T10:58:32.694 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4500 IOs, 1776845824 bytes 2023-10-11T10:58:32.816 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4600 IOs, 1802620928 bytes 2023-10-11T10:58:33.127 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4700 IOs, 1892409344 bytes 2023-10-11T10:58:33.349 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4800 IOs, 1957870592 bytes 2023-10-11T10:58:33.478 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4900 IOs, 1966761984 bytes 2023-10-11T10:58:33.659 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5000 IOs, 2014578688 bytes 2023-10-11T10:58:33.914 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5100 IOs, 2090135552 bytes 2023-10-11T10:58:34.110 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5200 IOs, 2135264256 bytes 2023-10-11T10:58:34.249 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5300 IOs, 2148261888 bytes 2023-10-11T10:58:34.447 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5400 IOs, 2203640832 bytes 2023-10-11T10:58:34.717 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5500 IOs, 2290092032 bytes 2023-10-11T10:58:34.882 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5600 IOs, 2324535296 bytes 2023-10-11T10:58:35.047 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5700 IOs, 2361784320 bytes 2023-10-11T10:58:35.220 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5800 IOs, 2410163200 bytes 2023-10-11T10:58:35.370 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5900 IOs, 2411515904 bytes 2023-10-11T10:58:35.549 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6000 IOs, 2448137216 bytes 2023-10-11T10:58:35.648 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6100 IOs, 2448737280 bytes 2023-10-11T10:58:35.895 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6200 IOs, 2522598400 bytes 2023-10-11T10:58:36.163 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6300 IOs, 2600087552 bytes 2023-10-11T10:58:36.302 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6400 IOs, 2620993536 bytes 2023-10-11T10:58:36.501 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6500 IOs, 2673374208 bytes 2023-10-11T10:58:36.513 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ error_code=124 2023-10-11T10:58:36.513 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ set -e 2023-10-11T10:58:36.513 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' 124 -eq 124 ']' 2023-10-11T10:58:36.514 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T10:58:36.514 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T10:58:36.514 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_27 2023-10-11T10:58:36.514 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ create_image_and_enable_mirror cluster2 mirror image_27 snapshot 128M 2023-10-11T10:58:36.514 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:58:36.515 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:58:36.515 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:58:36.515 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:58:36.515 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_27 2023-10-11T10:58:36.515 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:58:36.516 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local mode=snapshot 2023-10-11T10:58:36.516 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n snapshot ']' 2023-10-11T10:58:36.516 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:58:36.516 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ create_image cluster2 mirror image_27 128M 2023-10-11T10:58:36.516 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:58:36.517 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:58:36.517 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:58:36.517 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:58:36.517 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_27 2023-10-11T10:58:36.517 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:58:36.518 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local size=128 2023-10-11T10:58:36.518 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n 128M ']' 2023-10-11T10:58:36.518 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ size=128M 2023-10-11T10:58:36.518 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:58:36.518 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_27 2023-10-11T10:58:36.560 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' image = image ']' 2023-10-11T10:58:36.560 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ enable_mirror cluster2 mirror image_27 snapshot 2023-10-11T10:58:36.561 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:58:36.561 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:58:36.561 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_27 2023-10-11T10:58:36.561 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local mode=snapshot 2023-10-11T10:58:36.562 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_27 snapshot 2023-10-11T10:58:37.407 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:Mirroring enabled 2023-10-11T10:58:37.417 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 info mirror/image_27 2023-10-11T10:58:37.455 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:rbd image 'image_27': 2023-10-11T10:58:37.455 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: size 128 MiB in 32 objects 2023-10-11T10:58:37.455 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: order 22 (4 MiB objects) 2023-10-11T10:58:37.455 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: snapshot_count: 1 2023-10-11T10:58:37.456 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: id: 1346c0f8ef70 2023-10-11T10:58:37.456 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: block_name_prefix: rbd_data.1346c0f8ef70 2023-10-11T10:58:37.456 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: format: 2 2023-10-11T10:58:37.456 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-10-11T10:58:37.456 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: op_features: 2023-10-11T10:58:37.457 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: flags: 2023-10-11T10:58:37.457 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: create_timestamp: Wed Oct 11 10:58:36 2023 2023-10-11T10:58:37.457 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: access_timestamp: Wed Oct 11 10:58:36 2023 2023-10-11T10:58:37.457 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: modify_timestamp: Wed Oct 11 10:58:36 2023 2023-10-11T10:58:37.458 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring state: enabled 2023-10-11T10:58:37.458 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring mode: snapshot 2023-10-11T10:58:37.458 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring global id: fed85990-e790-4702-a20f-70d066da94cd 2023-10-11T10:58:37.458 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring primary: true 2023-10-11T10:58:37.459 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n '' ']' 2023-10-11T10:58:37.459 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ stress_write_image cluster2 mirror image_27 2023-10-11T10:58:37.460 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:58:37.460 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:58:37.460 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_27 2023-10-11T10:58:37.460 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-10-11T10:58:37.461 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local duration=8 2023-10-11T10:58:37.461 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ set +e 2023-10-11T10:58:37.461 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_27 --debug-rbd=20 --debug-journaler=20 2023-10-11T10:58:37.685 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:100 IOs, 19203072 bytes 2023-10-11T10:58:37.842 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:200 IOs, 48574464 bytes 2023-10-11T10:58:38.086 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:300 IOs, 142526464 bytes 2023-10-11T10:58:38.306 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:400 IOs, 190775296 bytes 2023-10-11T10:58:38.522 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:500 IOs, 241487872 bytes 2023-10-11T10:58:38.664 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:600 IOs, 262447104 bytes 2023-10-11T10:58:38.900 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:700 IOs, 332838912 bytes 2023-10-11T10:58:39.167 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:800 IOs, 425266176 bytes 2023-10-11T10:58:39.341 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:900 IOs, 476816384 bytes 2023-10-11T10:58:39.511 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1000 IOs, 494824448 bytes 2023-10-11T10:58:39.610 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1100 IOs, 495423488 bytes 2023-10-11T10:58:39.720 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1200 IOs, 496036864 bytes 2023-10-11T10:58:39.846 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1300 IOs, 528370688 bytes 2023-10-11T10:58:40.123 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1400 IOs, 601914368 bytes 2023-10-11T10:58:40.310 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1500 IOs, 641373184 bytes 2023-10-11T10:58:40.420 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1600 IOs, 651495424 bytes 2023-10-11T10:58:40.636 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1700 IOs, 722950144 bytes 2023-10-11T10:58:40.802 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1800 IOs, 754487296 bytes 2023-10-11T10:58:40.997 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1900 IOs, 830390272 bytes 2023-10-11T10:58:41.189 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2000 IOs, 850444288 bytes 2023-10-11T10:58:41.366 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2100 IOs, 888774656 bytes 2023-10-11T10:58:41.467 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2200 IOs, 889374720 bytes 2023-10-11T10:58:41.698 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2300 IOs, 952638464 bytes 2023-10-11T10:58:41.937 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2400 IOs, 1031785472 bytes 2023-10-11T10:58:42.218 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2500 IOs, 1067138048 bytes 2023-10-11T10:58:42.421 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2600 IOs, 1111816192 bytes 2023-10-11T10:58:42.659 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2700 IOs, 1187523584 bytes 2023-10-11T10:58:42.798 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2800 IOs, 1199250432 bytes 2023-10-11T10:58:42.955 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2900 IOs, 1226222592 bytes 2023-10-11T10:58:43.214 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3000 IOs, 1299721216 bytes 2023-10-11T10:58:43.390 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3100 IOs, 1336414208 bytes 2023-10-11T10:58:43.558 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3200 IOs, 1388246016 bytes 2023-10-11T10:58:43.804 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3300 IOs, 1460669440 bytes 2023-10-11T10:58:43.946 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3400 IOs, 1461311488 bytes 2023-10-11T10:58:44.149 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3500 IOs, 1518467072 bytes 2023-10-11T10:58:44.377 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3600 IOs, 1587584000 bytes 2023-10-11T10:58:44.582 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3700 IOs, 1651224576 bytes 2023-10-11T10:58:44.797 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3800 IOs, 1696174080 bytes 2023-10-11T10:58:44.913 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3900 IOs, 1702585344 bytes 2023-10-11T10:58:45.039 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4000 IOs, 1714703360 bytes 2023-10-11T10:58:45.231 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4100 IOs, 1776922624 bytes 2023-10-11T10:58:45.467 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ error_code=124 2023-10-11T10:58:45.467 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ set -e 2023-10-11T10:58:45.468 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' 124 -eq 124 ']' 2023-10-11T10:58:45.468 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T10:58:45.468 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T10:58:45.468 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_28 2023-10-11T10:58:45.468 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ create_image_and_enable_mirror cluster2 mirror image_28 snapshot 128M 2023-10-11T10:58:45.469 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:58:45.469 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:58:45.469 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:58:45.469 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:58:45.469 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_28 2023-10-11T10:58:45.469 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:58:45.470 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local mode=snapshot 2023-10-11T10:58:45.470 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n snapshot ']' 2023-10-11T10:58:45.470 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:58:45.470 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ create_image cluster2 mirror image_28 128M 2023-10-11T10:58:45.470 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:58:45.470 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:58:45.471 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:58:45.471 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:58:45.471 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_28 2023-10-11T10:58:45.471 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:58:45.471 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local size=128 2023-10-11T10:58:45.472 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n 128M ']' 2023-10-11T10:58:45.472 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ size=128M 2023-10-11T10:58:45.472 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:58:45.472 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_28 2023-10-11T10:58:45.517 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' image = image ']' 2023-10-11T10:58:45.517 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ enable_mirror cluster2 mirror image_28 snapshot 2023-10-11T10:58:45.517 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:58:45.518 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:58:45.518 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_28 2023-10-11T10:58:45.518 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local mode=snapshot 2023-10-11T10:58:45.518 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_28 snapshot 2023-10-11T10:58:46.402 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:Mirroring enabled 2023-10-11T10:58:46.413 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 info mirror/image_28 2023-10-11T10:58:46.452 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:rbd image 'image_28': 2023-10-11T10:58:46.452 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: size 128 MiB in 32 objects 2023-10-11T10:58:46.453 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: order 22 (4 MiB objects) 2023-10-11T10:58:46.453 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: snapshot_count: 1 2023-10-11T10:58:46.453 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: id: 13501036f91d 2023-10-11T10:58:46.453 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: block_name_prefix: rbd_data.13501036f91d 2023-10-11T10:58:46.453 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: format: 2 2023-10-11T10:58:46.454 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-10-11T10:58:46.454 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: op_features: 2023-10-11T10:58:46.454 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: flags: 2023-10-11T10:58:46.454 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: create_timestamp: Wed Oct 11 10:58:45 2023 2023-10-11T10:58:46.454 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: access_timestamp: Wed Oct 11 10:58:45 2023 2023-10-11T10:58:46.455 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: modify_timestamp: Wed Oct 11 10:58:45 2023 2023-10-11T10:58:46.455 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring state: enabled 2023-10-11T10:58:46.455 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring mode: snapshot 2023-10-11T10:58:46.455 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring global id: 48b201ee-f8ea-4845-97ba-749a1f35906b 2023-10-11T10:58:46.455 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring primary: true 2023-10-11T10:58:46.456 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n '' ']' 2023-10-11T10:58:46.456 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ stress_write_image cluster2 mirror image_28 2023-10-11T10:58:46.457 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:58:46.457 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:58:46.457 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_28 2023-10-11T10:58:46.457 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-10-11T10:58:46.458 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local duration=5 2023-10-11T10:58:46.458 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ set +e 2023-10-11T10:58:46.458 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_28 --debug-rbd=20 --debug-journaler=20 2023-10-11T10:58:46.745 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:100 IOs, 63906816 bytes 2023-10-11T10:58:46.904 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:200 IOs, 96834560 bytes 2023-10-11T10:58:47.003 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:300 IOs, 97431552 bytes 2023-10-11T10:58:47.106 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:400 IOs, 98048000 bytes 2023-10-11T10:58:47.280 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:500 IOs, 135075840 bytes 2023-10-11T10:58:47.428 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:600 IOs, 161070080 bytes 2023-10-11T10:58:47.602 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:700 IOs, 194272256 bytes 2023-10-11T10:58:47.837 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:800 IOs, 263299072 bytes 2023-10-11T10:58:48.018 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:900 IOs, 315789312 bytes 2023-10-11T10:58:48.173 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1000 IOs, 332394496 bytes 2023-10-11T10:58:48.343 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1100 IOs, 366791680 bytes 2023-10-11T10:58:48.528 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1200 IOs, 404972544 bytes 2023-10-11T10:58:48.711 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1300 IOs, 443362304 bytes 2023-10-11T10:58:48.903 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1400 IOs, 483644416 bytes 2023-10-11T10:58:49.154 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1500 IOs, 559341568 bytes 2023-10-11T10:58:49.348 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1600 IOs, 601134080 bytes 2023-10-11T10:58:49.540 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1700 IOs, 644240384 bytes 2023-10-11T10:58:49.671 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1800 IOs, 653270016 bytes 2023-10-11T10:58:49.895 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1900 IOs, 722304000 bytes 2023-10-11T10:58:50.133 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2000 IOs, 782714880 bytes 2023-10-11T10:58:50.336 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2100 IOs, 837950464 bytes 2023-10-11T10:58:50.528 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2200 IOs, 888128512 bytes 2023-10-11T10:58:50.639 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2300 IOs, 888757248 bytes 2023-10-11T10:58:50.862 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2400 IOs, 938994688 bytes 2023-10-11T10:58:50.993 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2500 IOs, 948820992 bytes 2023-10-11T10:58:51.146 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2600 IOs, 969111552 bytes 2023-10-11T10:58:51.356 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2700 IOs, 1023237120 bytes 2023-10-11T10:58:51.466 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ error_code=124 2023-10-11T10:58:51.466 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ set -e 2023-10-11T10:58:51.467 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' 124 -eq 124 ']' 2023-10-11T10:58:51.467 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T10:58:51.467 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T10:58:51.467 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_29 2023-10-11T10:58:51.468 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ create_image_and_enable_mirror cluster2 mirror image_29 snapshot 128M 2023-10-11T10:58:51.468 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:58:51.468 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:58:51.469 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:58:51.469 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:58:51.469 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_29 2023-10-11T10:58:51.469 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:58:51.470 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local mode=snapshot 2023-10-11T10:58:51.470 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n snapshot ']' 2023-10-11T10:58:51.470 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:58:51.471 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ create_image cluster2 mirror image_29 128M 2023-10-11T10:58:51.471 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:58:51.471 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:58:51.471 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:58:51.472 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:58:51.472 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_29 2023-10-11T10:58:51.472 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:58:51.473 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local size=128 2023-10-11T10:58:51.473 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n 128M ']' 2023-10-11T10:58:51.473 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ size=128M 2023-10-11T10:58:51.473 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:58:51.474 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_29 2023-10-11T10:58:51.514 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' image = image ']' 2023-10-11T10:58:51.514 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ enable_mirror cluster2 mirror image_29 snapshot 2023-10-11T10:58:51.514 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:58:51.515 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:58:51.515 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_29 2023-10-11T10:58:51.515 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local mode=snapshot 2023-10-11T10:58:51.515 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_29 snapshot 2023-10-11T10:58:52.652 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:Mirroring enabled 2023-10-11T10:58:52.664 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 info mirror/image_29 2023-10-11T10:58:52.703 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:rbd image 'image_29': 2023-10-11T10:58:52.703 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: size 128 MiB in 32 objects 2023-10-11T10:58:52.704 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: order 22 (4 MiB objects) 2023-10-11T10:58:52.704 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: snapshot_count: 1 2023-10-11T10:58:52.704 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: id: 13594357e54a 2023-10-11T10:58:52.704 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: block_name_prefix: rbd_data.13594357e54a 2023-10-11T10:58:52.704 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: format: 2 2023-10-11T10:58:52.704 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-10-11T10:58:52.705 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: op_features: 2023-10-11T10:58:52.705 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: flags: 2023-10-11T10:58:52.705 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: create_timestamp: Wed Oct 11 10:58:51 2023 2023-10-11T10:58:52.705 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: access_timestamp: Wed Oct 11 10:58:51 2023 2023-10-11T10:58:52.705 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: modify_timestamp: Wed Oct 11 10:58:51 2023 2023-10-11T10:58:52.706 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring state: enabled 2023-10-11T10:58:52.706 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring mode: snapshot 2023-10-11T10:58:52.706 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring global id: d589d227-75bc-45bf-a707-46d8d0068324 2023-10-11T10:58:52.706 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring primary: true 2023-10-11T10:58:52.707 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n '' ']' 2023-10-11T10:58:52.707 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ stress_write_image cluster2 mirror image_29 2023-10-11T10:58:52.707 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:58:52.708 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:58:52.708 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_29 2023-10-11T10:58:52.708 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-10-11T10:58:52.709 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local duration=10 2023-10-11T10:58:52.709 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ set +e 2023-10-11T10:58:52.709 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_29 --debug-rbd=20 --debug-journaler=20 2023-10-11T10:58:52.987 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:100 IOs, 50232320 bytes 2023-10-11T10:58:53.111 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:200 IOs, 60580864 bytes 2023-10-11T10:58:53.228 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:300 IOs, 67222528 bytes 2023-10-11T10:58:53.447 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:400 IOs, 143745024 bytes 2023-10-11T10:58:53.789 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:500 IOs, 264903680 bytes 2023-10-11T10:58:53.898 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:600 IOs, 272476160 bytes 2023-10-11T10:58:54.028 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:700 IOs, 286136320 bytes 2023-10-11T10:58:54.202 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:800 IOs, 314897408 bytes 2023-10-11T10:58:54.364 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:900 IOs, 357547008 bytes 2023-10-11T10:58:54.569 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1000 IOs, 400273408 bytes 2023-10-11T10:58:54.693 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1100 IOs, 411596800 bytes 2023-10-11T10:58:54.795 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1200 IOs, 413740032 bytes 2023-10-11T10:58:54.896 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1300 IOs, 414334976 bytes 2023-10-11T10:58:55.025 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1400 IOs, 437670912 bytes 2023-10-11T10:58:55.200 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1500 IOs, 463132672 bytes 2023-10-11T10:58:55.525 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1600 IOs, 568650752 bytes 2023-10-11T10:58:55.632 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1700 IOs, 569266176 bytes 2023-10-11T10:58:55.898 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1800 IOs, 665860096 bytes 2023-10-11T10:58:56.214 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1900 IOs, 764999680 bytes 2023-10-11T10:58:56.387 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2000 IOs, 800330752 bytes 2023-10-11T10:58:56.525 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2100 IOs, 821733376 bytes 2023-10-11T10:58:56.689 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2200 IOs, 858425344 bytes 2023-10-11T10:58:56.853 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2300 IOs, 889146368 bytes 2023-10-11T10:58:57.008 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2400 IOs, 917672960 bytes 2023-10-11T10:58:57.326 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2500 IOs, 1019973632 bytes 2023-10-11T10:58:57.577 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2600 IOs, 1074592768 bytes 2023-10-11T10:58:57.769 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2700 IOs, 1117429760 bytes 2023-10-11T10:58:57.992 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2800 IOs, 1180837888 bytes 2023-10-11T10:58:58.089 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2900 IOs, 1181448192 bytes 2023-10-11T10:58:58.234 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3000 IOs, 1206685696 bytes 2023-10-11T10:58:58.425 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3100 IOs, 1245049856 bytes 2023-10-11T10:58:58.679 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3200 IOs, 1327428608 bytes 2023-10-11T10:58:58.834 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3300 IOs, 1354281984 bytes 2023-10-11T10:58:59.004 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3400 IOs, 1389002752 bytes 2023-10-11T10:58:59.176 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3500 IOs, 1421844480 bytes 2023-10-11T10:58:59.311 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3600 IOs, 1439059968 bytes 2023-10-11T10:58:59.552 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3700 IOs, 1505514496 bytes 2023-10-11T10:58:59.730 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3800 IOs, 1543542784 bytes 2023-10-11T10:58:59.915 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3900 IOs, 1585478656 bytes 2023-10-11T10:59:00.068 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4000 IOs, 1610966016 bytes 2023-10-11T10:59:00.185 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4100 IOs, 1622175744 bytes 2023-10-11T10:59:00.305 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4200 IOs, 1628468224 bytes 2023-10-11T10:59:00.482 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4300 IOs, 1666599936 bytes 2023-10-11T10:59:00.634 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4400 IOs, 1692802048 bytes 2023-10-11T10:59:00.741 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4500 IOs, 1697599488 bytes 2023-10-11T10:59:00.946 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4600 IOs, 1744747520 bytes 2023-10-11T10:59:01.193 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4700 IOs, 1813531648 bytes 2023-10-11T10:59:01.342 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4800 IOs, 1840347136 bytes 2023-10-11T10:59:01.483 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4900 IOs, 1859034112 bytes 2023-10-11T10:59:01.613 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5000 IOs, 1888250880 bytes 2023-10-11T10:59:02.029 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5100 IOs, 1946635264 bytes 2023-10-11T10:59:02.199 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5200 IOs, 1980333056 bytes 2023-10-11T10:59:02.404 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5300 IOs, 2035701760 bytes 2023-10-11T10:59:02.555 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5400 IOs, 2066541568 bytes 2023-10-11T10:59:02.715 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ error_code=124 2023-10-11T10:59:02.715 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ set -e 2023-10-11T10:59:02.716 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' 124 -eq 124 ']' 2023-10-11T10:59:02.716 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T10:59:02.716 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T10:59:02.716 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_30 2023-10-11T10:59:02.717 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ create_image_and_enable_mirror cluster2 mirror image_30 snapshot 128M 2023-10-11T10:59:02.717 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:59:02.717 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:59:02.717 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:59:02.717 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:59:02.718 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_30 2023-10-11T10:59:02.718 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:59:02.718 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local mode=snapshot 2023-10-11T10:59:02.718 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n snapshot ']' 2023-10-11T10:59:02.718 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:59:02.719 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ create_image cluster2 mirror image_30 128M 2023-10-11T10:59:02.719 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:59:02.719 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:59:02.719 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:59:02.719 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:59:02.719 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_30 2023-10-11T10:59:02.720 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:59:02.720 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local size=128 2023-10-11T10:59:02.720 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n 128M ']' 2023-10-11T10:59:02.720 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ size=128M 2023-10-11T10:59:02.720 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:59:02.720 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_30 2023-10-11T10:59:02.764 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' image = image ']' 2023-10-11T10:59:02.764 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ enable_mirror cluster2 mirror image_30 snapshot 2023-10-11T10:59:02.764 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:59:02.765 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:59:02.765 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_30 2023-10-11T10:59:02.765 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local mode=snapshot 2023-10-11T10:59:02.765 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_30 snapshot 2023-10-11T10:59:03.396 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:Mirroring enabled 2023-10-11T10:59:03.405 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 info mirror/image_30 2023-10-11T10:59:03.444 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:rbd image 'image_30': 2023-10-11T10:59:03.444 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: size 128 MiB in 32 objects 2023-10-11T10:59:03.444 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: order 22 (4 MiB objects) 2023-10-11T10:59:03.444 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: snapshot_count: 1 2023-10-11T10:59:03.445 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: id: 1363d0724871 2023-10-11T10:59:03.445 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: block_name_prefix: rbd_data.1363d0724871 2023-10-11T10:59:03.445 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: format: 2 2023-10-11T10:59:03.445 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-10-11T10:59:03.445 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: op_features: 2023-10-11T10:59:03.446 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: flags: 2023-10-11T10:59:03.446 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: create_timestamp: Wed Oct 11 10:59:02 2023 2023-10-11T10:59:03.446 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: access_timestamp: Wed Oct 11 10:59:02 2023 2023-10-11T10:59:03.446 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: modify_timestamp: Wed Oct 11 10:59:02 2023 2023-10-11T10:59:03.446 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring state: enabled 2023-10-11T10:59:03.447 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring mode: snapshot 2023-10-11T10:59:03.447 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring global id: 3d66dfc7-26fa-4c9e-b7cc-ebb9eb40ba6a 2023-10-11T10:59:03.447 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring primary: true 2023-10-11T10:59:03.448 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n '' ']' 2023-10-11T10:59:03.448 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ stress_write_image cluster2 mirror image_30 2023-10-11T10:59:03.448 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:59:03.448 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:59:03.448 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_30 2023-10-11T10:59:03.449 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-10-11T10:59:03.449 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local duration=7 2023-10-11T10:59:03.449 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ set +e 2023-10-11T10:59:03.450 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_30 --debug-rbd=20 --debug-journaler=20 2023-10-11T10:59:03.773 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:100 IOs, 75318272 bytes 2023-10-11T10:59:03.997 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:200 IOs, 140015616 bytes 2023-10-11T10:59:04.242 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:300 IOs, 206237696 bytes 2023-10-11T10:59:04.387 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:400 IOs, 228925440 bytes 2023-10-11T10:59:04.527 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:500 IOs, 245805056 bytes 2023-10-11T10:59:04.546 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.7065445711404216) > weight (0.25) 2023-10-11T10:59:04.546 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.2904448964173387) > weight (0.25) 2023-10-11T10:59:04.546 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.7652773361849602) > weight (0.25) 2023-10-11T10:59:04.546 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.5558089685113176) > weight (0.25) 2023-10-11T10:59:04.547 INFO:tasks.rbd_mirror_thrash:waiting for 82 secs before thrashing 2023-10-11T10:59:04.738 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:600 IOs, 298285056 bytes 2023-10-11T10:59:04.995 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:700 IOs, 386460672 bytes 2023-10-11T10:59:05.203 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:800 IOs, 447125504 bytes 2023-10-11T10:59:05.382 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:900 IOs, 477244416 bytes 2023-10-11T10:59:05.605 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1000 IOs, 538456064 bytes 2023-10-11T10:59:05.796 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1100 IOs, 578341888 bytes 2023-10-11T10:59:05.955 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1200 IOs, 604921856 bytes 2023-10-11T10:59:06.089 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1300 IOs, 628054016 bytes 2023-10-11T10:59:06.273 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1400 IOs, 663433216 bytes 2023-10-11T10:59:06.458 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1500 IOs, 708668416 bytes 2023-10-11T10:59:06.713 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1600 IOs, 799901696 bytes 2023-10-11T10:59:06.931 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1700 IOs, 836239360 bytes 2023-10-11T10:59:07.074 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1800 IOs, 860998656 bytes 2023-10-11T10:59:07.324 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1900 IOs, 940137472 bytes 2023-10-11T10:59:07.532 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2000 IOs, 990123008 bytes 2023-10-11T10:59:07.646 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2100 IOs, 1005256704 bytes 2023-10-11T10:59:07.806 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2200 IOs, 1018815488 bytes 2023-10-11T10:59:07.957 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2300 IOs, 1041027072 bytes 2023-10-11T10:59:08.157 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2400 IOs, 1086203904 bytes 2023-10-11T10:59:08.346 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2500 IOs, 1133368320 bytes 2023-10-11T10:59:08.589 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2600 IOs, 1202062336 bytes 2023-10-11T10:59:08.718 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2700 IOs, 1217700864 bytes 2023-10-11T10:59:08.926 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2800 IOs, 1269440512 bytes 2023-10-11T10:59:09.085 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2900 IOs, 1301947392 bytes 2023-10-11T10:59:09.281 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3000 IOs, 1356097536 bytes 2023-10-11T10:59:09.396 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3100 IOs, 1356703744 bytes 2023-10-11T10:59:09.598 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3200 IOs, 1403309056 bytes 2023-10-11T10:59:09.784 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3300 IOs, 1460967424 bytes 2023-10-11T10:59:09.965 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3400 IOs, 1480682496 bytes 2023-10-11T10:59:10.272 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3500 IOs, 1586083840 bytes 2023-10-11T10:59:10.455 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ error_code=124 2023-10-11T10:59:10.455 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ set -e 2023-10-11T10:59:10.455 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' 124 -eq 124 ']' 2023-10-11T10:59:10.456 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T10:59:10.456 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T10:59:10.456 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_31 2023-10-11T10:59:10.456 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ create_image_and_enable_mirror cluster2 mirror image_31 snapshot 128M 2023-10-11T10:59:10.457 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:59:10.457 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:59:10.457 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:59:10.457 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:59:10.458 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_31 2023-10-11T10:59:10.458 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:59:10.458 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local mode=snapshot 2023-10-11T10:59:10.458 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n snapshot ']' 2023-10-11T10:59:10.459 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:59:10.459 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ create_image cluster2 mirror image_31 128M 2023-10-11T10:59:10.459 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:59:10.460 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:59:10.460 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:59:10.460 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:59:10.460 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_31 2023-10-11T10:59:10.461 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:59:10.461 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local size=128 2023-10-11T10:59:10.461 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n 128M ']' 2023-10-11T10:59:10.462 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ size=128M 2023-10-11T10:59:10.462 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:59:10.462 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_31 2023-10-11T10:59:10.503 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' image = image ']' 2023-10-11T10:59:10.503 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ enable_mirror cluster2 mirror image_31 snapshot 2023-10-11T10:59:10.503 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:59:10.503 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:59:10.504 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_31 2023-10-11T10:59:10.504 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local mode=snapshot 2023-10-11T10:59:10.504 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_31 snapshot 2023-10-11T10:59:11.253 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:Mirroring enabled 2023-10-11T10:59:11.264 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 info mirror/image_31 2023-10-11T10:59:11.302 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:rbd image 'image_31': 2023-10-11T10:59:11.303 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: size 128 MiB in 32 objects 2023-10-11T10:59:11.303 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: order 22 (4 MiB objects) 2023-10-11T10:59:11.303 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: snapshot_count: 1 2023-10-11T10:59:11.303 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: id: 136b8f41d743 2023-10-11T10:59:11.304 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: block_name_prefix: rbd_data.136b8f41d743 2023-10-11T10:59:11.304 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: format: 2 2023-10-11T10:59:11.304 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-10-11T10:59:11.304 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: op_features: 2023-10-11T10:59:11.305 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: flags: 2023-10-11T10:59:11.305 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: create_timestamp: Wed Oct 11 10:59:10 2023 2023-10-11T10:59:11.305 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: access_timestamp: Wed Oct 11 10:59:10 2023 2023-10-11T10:59:11.305 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: modify_timestamp: Wed Oct 11 10:59:10 2023 2023-10-11T10:59:11.306 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring state: enabled 2023-10-11T10:59:11.306 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring mode: snapshot 2023-10-11T10:59:11.306 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring global id: da5623f9-4467-416e-af1d-0ee95e25c12e 2023-10-11T10:59:11.307 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring primary: true 2023-10-11T10:59:11.308 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n '' ']' 2023-10-11T10:59:11.308 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ stress_write_image cluster2 mirror image_31 2023-10-11T10:59:11.309 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:59:11.309 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:59:11.309 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_31 2023-10-11T10:59:11.309 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-10-11T10:59:11.310 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local duration=9 2023-10-11T10:59:11.310 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ set +e 2023-10-11T10:59:11.310 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_31 --debug-rbd=20 --debug-journaler=20 2023-10-11T10:59:11.626 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:100 IOs, 85486592 bytes 2023-10-11T10:59:11.807 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:200 IOs, 111657984 bytes 2023-10-11T10:59:12.040 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:300 IOs, 179692544 bytes 2023-10-11T10:59:12.234 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:400 IOs, 247960576 bytes 2023-10-11T10:59:12.493 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:500 IOs, 302539776 bytes 2023-10-11T10:59:12.708 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:600 IOs, 364370944 bytes 2023-10-11T10:59:12.884 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:700 IOs, 392563712 bytes 2023-10-11T10:59:13.073 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:800 IOs, 416349184 bytes 2023-10-11T10:59:13.248 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:900 IOs, 461219840 bytes 2023-10-11T10:59:13.405 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1000 IOs, 486700032 bytes 2023-10-11T10:59:13.608 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1100 IOs, 511394816 bytes 2023-10-11T10:59:13.738 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1200 IOs, 520992768 bytes 2023-10-11T10:59:13.857 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1300 IOs, 534031360 bytes 2023-10-11T10:59:14.078 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1400 IOs, 590825472 bytes 2023-10-11T10:59:14.311 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1500 IOs, 659415040 bytes 2023-10-11T10:59:14.465 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1600 IOs, 689024000 bytes 2023-10-11T10:59:14.806 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1700 IOs, 808224768 bytes 2023-10-11T10:59:14.955 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1800 IOs, 830219264 bytes 2023-10-11T10:59:15.210 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1900 IOs, 902388736 bytes 2023-10-11T10:59:15.330 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2000 IOs, 911429632 bytes 2023-10-11T10:59:15.482 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2100 IOs, 931770368 bytes 2023-10-11T10:59:15.557 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.6239252654974712) > weight (0.25) 2023-10-11T10:59:15.557 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.7314869401417979) > weight (0.25) 2023-10-11T10:59:15.558 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.643482859135832) > weight (0.25) 2023-10-11T10:59:15.558 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.6006415849196213) > weight (0.25) 2023-10-11T10:59:15.558 INFO:tasks.rbd_mirror_thrash:waiting for 87 secs before thrashing 2023-10-11T10:59:15.691 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2200 IOs, 993902592 bytes 2023-10-11T10:59:15.928 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2300 IOs, 1054931968 bytes 2023-10-11T10:59:15.983 INFO:tasks.ceph.cluster2.osd.2.smithi111.stderr:2023-10-11T10:59:15.982+0000 7fa35bd9b700 -1 reset not still connected to 0x5564047fa5b0 2023-10-11T10:59:16.047 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2400 IOs, 1061213184 bytes 2023-10-11T10:59:16.295 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2500 IOs, 1140457472 bytes 2023-10-11T10:59:16.399 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2600 IOs, 1144574976 bytes 2023-10-11T10:59:16.622 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2700 IOs, 1209344000 bytes 2023-10-11T10:59:16.895 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2800 IOs, 1297875968 bytes 2023-10-11T10:59:17.085 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2900 IOs, 1341151232 bytes 2023-10-11T10:59:17.350 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3000 IOs, 1429860352 bytes 2023-10-11T10:59:17.516 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3100 IOs, 1460440064 bytes 2023-10-11T10:59:17.749 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3200 IOs, 1527166976 bytes 2023-10-11T10:59:17.896 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3300 IOs, 1546907648 bytes 2023-10-11T10:59:18.154 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3400 IOs, 1640801280 bytes 2023-10-11T10:59:18.321 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3500 IOs, 1666289664 bytes 2023-10-11T10:59:18.555 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3600 IOs, 1743631360 bytes 2023-10-11T10:59:18.766 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3700 IOs, 1791014912 bytes 2023-10-11T10:59:18.964 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3800 IOs, 1839489024 bytes 2023-10-11T10:59:19.138 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3900 IOs, 1871910912 bytes 2023-10-11T10:59:19.263 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4000 IOs, 1898835968 bytes 2023-10-11T10:59:19.438 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4100 IOs, 1934743552 bytes 2023-10-11T10:59:19.547 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4200 IOs, 1944776704 bytes 2023-10-11T10:59:19.830 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4300 IOs, 2051289088 bytes 2023-10-11T10:59:20.037 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4400 IOs, 2090421248 bytes 2023-10-11T10:59:20.140 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4500 IOs, 2091331584 bytes 2023-10-11T10:59:20.254 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4600 IOs, 2095483904 bytes 2023-10-11T10:59:20.316 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ error_code=124 2023-10-11T10:59:20.316 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ set -e 2023-10-11T10:59:20.316 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' 124 -eq 124 ']' 2023-10-11T10:59:20.316 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T10:59:20.317 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T10:59:20.317 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_32 2023-10-11T10:59:20.317 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ create_image_and_enable_mirror cluster2 mirror image_32 snapshot 128M 2023-10-11T10:59:20.317 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:59:20.317 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:59:20.318 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:59:20.318 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:59:20.318 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_32 2023-10-11T10:59:20.318 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:59:20.318 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local mode=snapshot 2023-10-11T10:59:20.319 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n snapshot ']' 2023-10-11T10:59:20.319 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:59:20.319 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ create_image cluster2 mirror image_32 128M 2023-10-11T10:59:20.319 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:59:20.319 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:59:20.320 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:59:20.320 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:59:20.320 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_32 2023-10-11T10:59:20.320 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:59:20.320 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local size=128 2023-10-11T10:59:20.321 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n 128M ']' 2023-10-11T10:59:20.321 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ size=128M 2023-10-11T10:59:20.321 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:59:20.321 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_32 2023-10-11T10:59:20.366 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' image = image ']' 2023-10-11T10:59:20.366 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ enable_mirror cluster2 mirror image_32 snapshot 2023-10-11T10:59:20.367 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:59:20.367 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:59:20.367 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_32 2023-10-11T10:59:20.367 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local mode=snapshot 2023-10-11T10:59:20.367 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_32 snapshot 2023-10-11T10:59:21.421 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:Mirroring enabled 2023-10-11T10:59:21.433 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 info mirror/image_32 2023-10-11T10:59:21.469 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:rbd image 'image_32': 2023-10-11T10:59:21.470 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: size 128 MiB in 32 objects 2023-10-11T10:59:21.470 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: order 22 (4 MiB objects) 2023-10-11T10:59:21.470 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: snapshot_count: 1 2023-10-11T10:59:21.470 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: id: 1373b20f3021 2023-10-11T10:59:21.470 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: block_name_prefix: rbd_data.1373b20f3021 2023-10-11T10:59:21.471 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: format: 2 2023-10-11T10:59:21.471 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-10-11T10:59:21.471 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: op_features: 2023-10-11T10:59:21.471 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: flags: 2023-10-11T10:59:21.471 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: create_timestamp: Wed Oct 11 10:59:20 2023 2023-10-11T10:59:21.472 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: access_timestamp: Wed Oct 11 10:59:20 2023 2023-10-11T10:59:21.472 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: modify_timestamp: Wed Oct 11 10:59:20 2023 2023-10-11T10:59:21.472 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring state: enabled 2023-10-11T10:59:21.472 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring mode: snapshot 2023-10-11T10:59:21.472 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring global id: b4fc3ca8-51a4-45f4-a987-97af54ee49dc 2023-10-11T10:59:21.473 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring primary: true 2023-10-11T10:59:21.473 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n '' ']' 2023-10-11T10:59:21.474 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ stress_write_image cluster2 mirror image_32 2023-10-11T10:59:21.474 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:59:21.474 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:59:21.474 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_32 2023-10-11T10:59:21.475 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-10-11T10:59:21.475 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local duration=5 2023-10-11T10:59:21.475 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ set +e 2023-10-11T10:59:21.476 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_32 --debug-rbd=20 --debug-journaler=20 2023-10-11T10:59:21.661 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:100 IOs, 3219456 bytes 2023-10-11T10:59:21.855 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:200 IOs, 45973504 bytes 2023-10-11T10:59:21.994 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:300 IOs, 70659072 bytes 2023-10-11T10:59:22.129 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:400 IOs, 85441536 bytes 2023-10-11T10:59:22.349 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:500 IOs, 144262144 bytes 2023-10-11T10:59:22.516 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:600 IOs, 176357376 bytes 2023-10-11T10:59:22.734 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:700 IOs, 219581440 bytes 2023-10-11T10:59:22.948 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:800 IOs, 268397568 bytes 2023-10-11T10:59:23.186 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:900 IOs, 340322304 bytes 2023-10-11T10:59:23.316 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1000 IOs, 374737920 bytes 2023-10-11T10:59:23.505 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1100 IOs, 400060416 bytes 2023-10-11T10:59:23.729 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1200 IOs, 460618752 bytes 2023-10-11T10:59:23.870 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1300 IOs, 487733248 bytes 2023-10-11T10:59:24.075 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1400 IOs, 538423296 bytes 2023-10-11T10:59:24.218 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1500 IOs, 566370304 bytes 2023-10-11T10:59:24.435 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1600 IOs, 611506176 bytes 2023-10-11T10:59:24.582 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1700 IOs, 635694080 bytes 2023-10-11T10:59:24.872 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1800 IOs, 734075904 bytes 2023-10-11T10:59:25.028 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1900 IOs, 765585408 bytes 2023-10-11T10:59:25.226 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2000 IOs, 800972800 bytes 2023-10-11T10:59:25.358 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2100 IOs, 823408640 bytes 2023-10-11T10:59:25.565 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2200 IOs, 891960320 bytes 2023-10-11T10:59:25.758 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2300 IOs, 925914112 bytes 2023-10-11T10:59:26.033 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2400 IOs, 1009810432 bytes 2023-10-11T10:59:26.166 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2500 IOs, 1027045376 bytes 2023-10-11T10:59:26.419 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2600 IOs, 1091852288 bytes 2023-10-11T10:59:26.480 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ error_code=124 2023-10-11T10:59:26.480 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ set -e 2023-10-11T10:59:26.480 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' 124 -eq 124 ']' 2023-10-11T10:59:26.480 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T10:59:26.480 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T10:59:26.481 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_33 2023-10-11T10:59:26.481 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ create_image_and_enable_mirror cluster2 mirror image_33 snapshot 128M 2023-10-11T10:59:26.481 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:59:26.481 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:59:26.481 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:59:26.481 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:59:26.482 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_33 2023-10-11T10:59:26.482 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:59:26.482 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local mode=snapshot 2023-10-11T10:59:26.482 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n snapshot ']' 2023-10-11T10:59:26.482 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:59:26.483 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ create_image cluster2 mirror image_33 128M 2023-10-11T10:59:26.483 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:59:26.483 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:59:26.483 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:59:26.483 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:59:26.483 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_33 2023-10-11T10:59:26.484 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:59:26.484 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local size=128 2023-10-11T10:59:26.484 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n 128M ']' 2023-10-11T10:59:26.484 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ size=128M 2023-10-11T10:59:26.484 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:59:26.484 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_33 2023-10-11T10:59:26.528 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' image = image ']' 2023-10-11T10:59:26.528 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ enable_mirror cluster2 mirror image_33 snapshot 2023-10-11T10:59:26.529 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:59:26.529 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:59:26.529 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_33 2023-10-11T10:59:26.529 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local mode=snapshot 2023-10-11T10:59:26.529 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_33 snapshot 2023-10-11T10:59:27.448 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:Mirroring enabled 2023-10-11T10:59:27.458 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 info mirror/image_33 2023-10-11T10:59:27.494 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:rbd image 'image_33': 2023-10-11T10:59:27.494 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: size 128 MiB in 32 objects 2023-10-11T10:59:27.495 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: order 22 (4 MiB objects) 2023-10-11T10:59:27.495 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: snapshot_count: 1 2023-10-11T10:59:27.495 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: id: 137c1fd9b85b 2023-10-11T10:59:27.495 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: block_name_prefix: rbd_data.137c1fd9b85b 2023-10-11T10:59:27.495 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: format: 2 2023-10-11T10:59:27.496 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-10-11T10:59:27.496 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: op_features: 2023-10-11T10:59:27.496 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: flags: 2023-10-11T10:59:27.496 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: create_timestamp: Wed Oct 11 10:59:26 2023 2023-10-11T10:59:27.496 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: access_timestamp: Wed Oct 11 10:59:26 2023 2023-10-11T10:59:27.497 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: modify_timestamp: Wed Oct 11 10:59:26 2023 2023-10-11T10:59:27.497 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring state: enabled 2023-10-11T10:59:27.497 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring mode: snapshot 2023-10-11T10:59:27.497 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring global id: a8456df7-6856-4049-9b99-fded34440c1f 2023-10-11T10:59:27.497 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring primary: true 2023-10-11T10:59:27.498 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n '' ']' 2023-10-11T10:59:27.498 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ stress_write_image cluster2 mirror image_33 2023-10-11T10:59:27.499 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:59:27.499 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:59:27.499 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_33 2023-10-11T10:59:27.499 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-10-11T10:59:27.500 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local duration=6 2023-10-11T10:59:27.501 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ set +e 2023-10-11T10:59:27.501 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_33 --debug-rbd=20 --debug-journaler=20 2023-10-11T10:59:27.842 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:100 IOs, 86979584 bytes 2023-10-11T10:59:28.101 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:200 IOs, 164924416 bytes 2023-10-11T10:59:28.327 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:300 IOs, 222986240 bytes 2023-10-11T10:59:28.514 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:400 IOs, 263131136 bytes 2023-10-11T10:59:28.705 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:500 IOs, 307176448 bytes 2023-10-11T10:59:28.854 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:600 IOs, 327673856 bytes 2023-10-11T10:59:28.962 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:700 IOs, 328299520 bytes 2023-10-11T10:59:29.171 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:800 IOs, 381052928 bytes 2023-10-11T10:59:29.597 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:900 IOs, 454797312 bytes 2023-10-11T10:59:29.743 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1000 IOs, 468943872 bytes 2023-10-11T10:59:29.966 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1100 IOs, 529338368 bytes 2023-10-11T10:59:30.093 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1200 IOs, 538077184 bytes 2023-10-11T10:59:30.237 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1300 IOs, 557919232 bytes 2023-10-11T10:59:30.389 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1400 IOs, 582065152 bytes 2023-10-11T10:59:30.595 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1500 IOs, 626828288 bytes 2023-10-11T10:59:30.813 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1600 IOs, 683703296 bytes 2023-10-11T10:59:31.019 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1700 IOs, 749012992 bytes 2023-10-11T10:59:31.254 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1800 IOs, 802322432 bytes 2023-10-11T10:59:31.516 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1900 IOs, 875408384 bytes 2023-10-11T10:59:31.630 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2000 IOs, 879686656 bytes 2023-10-11T10:59:31.767 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2100 IOs, 896696320 bytes 2023-10-11T10:59:31.987 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2200 IOs, 952483840 bytes 2023-10-11T10:59:32.154 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2300 IOs, 987228160 bytes 2023-10-11T10:59:32.323 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2400 IOs, 1020961792 bytes 2023-10-11T10:59:32.624 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2500 IOs, 1122457600 bytes 2023-10-11T10:59:32.769 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2600 IOs, 1142670336 bytes 2023-10-11T10:59:32.937 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2700 IOs, 1177619456 bytes 2023-10-11T10:59:33.094 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2800 IOs, 1203594240 bytes 2023-10-11T10:59:33.298 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2900 IOs, 1247098880 bytes 2023-10-11T10:59:33.408 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3000 IOs, 1247725568 bytes 2023-10-11T10:59:33.506 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ error_code=124 2023-10-11T10:59:33.506 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ set -e 2023-10-11T10:59:33.506 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' 124 -eq 124 ']' 2023-10-11T10:59:33.507 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T10:59:33.507 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T10:59:33.507 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_34 2023-10-11T10:59:33.507 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ create_image_and_enable_mirror cluster2 mirror image_34 snapshot 128M 2023-10-11T10:59:33.507 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:59:33.508 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:59:33.508 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:59:33.508 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:59:33.508 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_34 2023-10-11T10:59:33.508 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:59:33.509 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local mode=snapshot 2023-10-11T10:59:33.509 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n snapshot ']' 2023-10-11T10:59:33.509 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:59:33.509 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ create_image cluster2 mirror image_34 128M 2023-10-11T10:59:33.509 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:59:33.510 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:59:33.510 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:59:33.510 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:59:33.510 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_34 2023-10-11T10:59:33.510 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:59:33.511 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local size=128 2023-10-11T10:59:33.511 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n 128M ']' 2023-10-11T10:59:33.511 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ size=128M 2023-10-11T10:59:33.511 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:59:33.511 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_34 2023-10-11T10:59:33.554 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' image = image ']' 2023-10-11T10:59:33.554 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ enable_mirror cluster2 mirror image_34 snapshot 2023-10-11T10:59:33.555 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:59:33.555 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:59:33.555 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_34 2023-10-11T10:59:33.555 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local mode=snapshot 2023-10-11T10:59:33.556 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_34 snapshot 2023-10-11T10:59:34.405 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:Mirroring enabled 2023-10-11T10:59:34.416 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 info mirror/image_34 2023-10-11T10:59:34.453 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:rbd image 'image_34': 2023-10-11T10:59:34.453 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: size 128 MiB in 32 objects 2023-10-11T10:59:34.454 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: order 22 (4 MiB objects) 2023-10-11T10:59:34.454 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: snapshot_count: 1 2023-10-11T10:59:34.454 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: id: 1384b6a61232 2023-10-11T10:59:34.454 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: block_name_prefix: rbd_data.1384b6a61232 2023-10-11T10:59:34.455 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: format: 2 2023-10-11T10:59:34.455 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-10-11T10:59:34.455 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: op_features: 2023-10-11T10:59:34.455 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: flags: 2023-10-11T10:59:34.456 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: create_timestamp: Wed Oct 11 10:59:33 2023 2023-10-11T10:59:34.456 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: access_timestamp: Wed Oct 11 10:59:33 2023 2023-10-11T10:59:34.456 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: modify_timestamp: Wed Oct 11 10:59:33 2023 2023-10-11T10:59:34.456 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring state: enabled 2023-10-11T10:59:34.457 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring mode: snapshot 2023-10-11T10:59:34.457 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring global id: 78a70dcb-6515-4835-bfd0-d59363f95aba 2023-10-11T10:59:34.457 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring primary: true 2023-10-11T10:59:34.458 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n '' ']' 2023-10-11T10:59:34.459 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ stress_write_image cluster2 mirror image_34 2023-10-11T10:59:34.459 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:59:34.459 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:59:34.459 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_34 2023-10-11T10:59:34.460 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-10-11T10:59:34.460 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local duration=12 2023-10-11T10:59:34.460 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ set +e 2023-10-11T10:59:34.460 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_34 --debug-rbd=20 --debug-journaler=20 2023-10-11T10:59:34.731 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:100 IOs, 51949568 bytes 2023-10-11T10:59:34.846 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:200 IOs, 59411456 bytes 2023-10-11T10:59:35.072 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:300 IOs, 132079616 bytes 2023-10-11T10:59:35.211 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:400 IOs, 148731904 bytes 2023-10-11T10:59:35.335 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:500 IOs, 159245312 bytes 2023-10-11T10:59:35.474 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:600 IOs, 185204736 bytes 2023-10-11T10:59:35.676 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:700 IOs, 242188288 bytes 2023-10-11T10:59:35.826 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:800 IOs, 264470528 bytes 2023-10-11T10:59:36.120 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:900 IOs, 354641920 bytes 2023-10-11T10:59:36.367 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1000 IOs, 425843712 bytes 2023-10-11T10:59:36.489 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1100 IOs, 440140800 bytes 2023-10-11T10:59:36.893 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1200 IOs, 494286848 bytes 2023-10-11T10:59:37.029 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1300 IOs, 511998976 bytes 2023-10-11T10:59:37.249 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1400 IOs, 586250240 bytes 2023-10-11T10:59:37.480 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1500 IOs, 643030016 bytes 2023-10-11T10:59:37.672 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1600 IOs, 687623168 bytes 2023-10-11T10:59:37.835 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1700 IOs, 724204544 bytes 2023-10-11T10:59:38.020 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1800 IOs, 765842432 bytes 2023-10-11T10:59:38.155 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1900 IOs, 798193664 bytes 2023-10-11T10:59:38.322 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2000 IOs, 815524864 bytes 2023-10-11T10:59:38.524 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2100 IOs, 870352896 bytes 2023-10-11T10:59:38.641 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2200 IOs, 878160896 bytes 2023-10-11T10:59:38.805 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2300 IOs, 921667584 bytes 2023-10-11T10:59:38.973 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2400 IOs, 939051008 bytes 2023-10-11T10:59:39.075 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2500 IOs, 939646976 bytes 2023-10-11T10:59:39.197 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2600 IOs, 946381824 bytes 2023-10-11T10:59:39.316 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2700 IOs, 953981952 bytes 2023-10-11T10:59:39.568 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2800 IOs, 1028920320 bytes 2023-10-11T10:59:39.727 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2900 IOs, 1056622592 bytes 2023-10-11T10:59:39.866 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3000 IOs, 1072104448 bytes 2023-10-11T10:59:40.158 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3100 IOs, 1166006272 bytes 2023-10-11T10:59:40.360 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3200 IOs, 1213634560 bytes 2023-10-11T10:59:40.521 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3300 IOs, 1244125184 bytes 2023-10-11T10:59:40.700 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3400 IOs, 1284176896 bytes 2023-10-11T10:59:40.833 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3500 IOs, 1295266816 bytes 2023-10-11T10:59:41.124 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3600 IOs, 1390127104 bytes 2023-10-11T10:59:41.361 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3700 IOs, 1465535488 bytes 2023-10-11T10:59:41.472 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3800 IOs, 1466180608 bytes 2023-10-11T10:59:41.692 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3900 IOs, 1524357120 bytes 2023-10-11T10:59:41.862 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4000 IOs, 1560518656 bytes 2023-10-11T10:59:42.199 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4100 IOs, 1633286144 bytes 2023-10-11T10:59:42.456 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4200 IOs, 1721544704 bytes 2023-10-11T10:59:42.821 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4300 IOs, 1820238848 bytes 2023-10-11T10:59:43.064 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4400 IOs, 1889654784 bytes 2023-10-11T10:59:43.258 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4500 IOs, 1932167168 bytes 2023-10-11T10:59:43.485 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4600 IOs, 1994024960 bytes 2023-10-11T10:59:43.673 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4700 IOs, 2048732160 bytes 2023-10-11T10:59:43.803 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4800 IOs, 2049356800 bytes 2023-10-11T10:59:43.992 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4900 IOs, 2095887360 bytes 2023-10-11T10:59:44.239 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5000 IOs, 2158429184 bytes 2023-10-11T10:59:44.426 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5100 IOs, 2204347392 bytes 2023-10-11T10:59:44.622 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5200 IOs, 2247337984 bytes 2023-10-11T10:59:44.756 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5300 IOs, 2260923392 bytes 2023-10-11T10:59:44.921 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5400 IOs, 2316261376 bytes 2023-10-11T10:59:45.098 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5500 IOs, 2331164672 bytes 2023-10-11T10:59:45.267 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5600 IOs, 2362211328 bytes 2023-10-11T10:59:45.409 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5700 IOs, 2398745600 bytes 2023-10-11T10:59:45.629 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5800 IOs, 2459570176 bytes 2023-10-11T10:59:45.758 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5900 IOs, 2466590720 bytes 2023-10-11T10:59:45.924 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6000 IOs, 2495585280 bytes 2023-10-11T10:59:46.157 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6100 IOs, 2553695232 bytes 2023-10-11T10:59:46.463 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ error_code=124 2023-10-11T10:59:46.463 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ set -e 2023-10-11T10:59:46.463 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' 124 -eq 124 ']' 2023-10-11T10:59:46.463 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T10:59:46.464 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T10:59:46.464 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_35 2023-10-11T10:59:46.464 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ create_image_and_enable_mirror cluster2 mirror image_35 snapshot 128M 2023-10-11T10:59:46.464 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:59:46.464 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:59:46.464 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:59:46.465 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:59:46.465 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_35 2023-10-11T10:59:46.465 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:59:46.465 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local mode=snapshot 2023-10-11T10:59:46.465 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n snapshot ']' 2023-10-11T10:59:46.465 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:59:46.466 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ create_image cluster2 mirror image_35 128M 2023-10-11T10:59:46.466 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:59:46.466 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:59:46.466 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:59:46.466 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:59:46.467 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_35 2023-10-11T10:59:46.467 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:59:46.467 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local size=128 2023-10-11T10:59:46.467 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n 128M ']' 2023-10-11T10:59:46.467 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ size=128M 2023-10-11T10:59:46.467 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:59:46.468 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_35 2023-10-11T10:59:46.513 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' image = image ']' 2023-10-11T10:59:46.513 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ enable_mirror cluster2 mirror image_35 snapshot 2023-10-11T10:59:46.513 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:59:46.513 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:59:46.514 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_35 2023-10-11T10:59:46.514 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local mode=snapshot 2023-10-11T10:59:46.514 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_35 snapshot 2023-10-11T10:59:47.417 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:Mirroring enabled 2023-10-11T10:59:47.428 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 info mirror/image_35 2023-10-11T10:59:47.465 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:rbd image 'image_35': 2023-10-11T10:59:47.465 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: size 128 MiB in 32 objects 2023-10-11T10:59:47.465 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: order 22 (4 MiB objects) 2023-10-11T10:59:47.465 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: snapshot_count: 1 2023-10-11T10:59:47.465 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: id: 138c5ed36144 2023-10-11T10:59:47.466 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: block_name_prefix: rbd_data.138c5ed36144 2023-10-11T10:59:47.466 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: format: 2 2023-10-11T10:59:47.466 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-10-11T10:59:47.466 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: op_features: 2023-10-11T10:59:47.466 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: flags: 2023-10-11T10:59:47.467 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: create_timestamp: Wed Oct 11 10:59:46 2023 2023-10-11T10:59:47.467 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: access_timestamp: Wed Oct 11 10:59:46 2023 2023-10-11T10:59:47.467 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: modify_timestamp: Wed Oct 11 10:59:46 2023 2023-10-11T10:59:47.467 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring state: enabled 2023-10-11T10:59:47.467 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring mode: snapshot 2023-10-11T10:59:47.468 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring global id: d4ebf971-35de-4686-a8a6-e84c09701f07 2023-10-11T10:59:47.468 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring primary: true 2023-10-11T10:59:47.468 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n '' ']' 2023-10-11T10:59:47.469 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ stress_write_image cluster2 mirror image_35 2023-10-11T10:59:47.469 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:59:47.469 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:59:47.469 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_35 2023-10-11T10:59:47.469 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-10-11T10:59:47.470 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local duration=9 2023-10-11T10:59:47.470 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ set +e 2023-10-11T10:59:47.471 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_35 --debug-rbd=20 --debug-journaler=20 2023-10-11T10:59:47.763 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:100 IOs, 66334720 bytes 2023-10-11T10:59:48.021 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:200 IOs, 136433664 bytes 2023-10-11T10:59:48.216 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:300 IOs, 180904960 bytes 2023-10-11T10:59:48.386 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:400 IOs, 217119744 bytes 2023-10-11T10:59:48.514 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:500 IOs, 226508800 bytes 2023-10-11T10:59:48.635 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:600 IOs, 239149056 bytes 2023-10-11T10:59:48.833 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:700 IOs, 296808448 bytes 2023-10-11T10:59:49.019 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:800 IOs, 338085888 bytes 2023-10-11T10:59:49.173 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:900 IOs, 369748992 bytes 2023-10-11T10:59:49.285 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1000 IOs, 375640064 bytes 2023-10-11T10:59:49.447 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1100 IOs, 401549312 bytes 2023-10-11T10:59:49.736 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1200 IOs, 489681920 bytes 2023-10-11T10:59:49.882 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1300 IOs, 515846144 bytes 2023-10-11T10:59:50.043 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1400 IOs, 547118080 bytes 2023-10-11T10:59:50.187 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1500 IOs, 565415936 bytes 2023-10-11T10:59:50.355 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1600 IOs, 592092160 bytes 2023-10-11T10:59:50.466 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1700 IOs, 593949696 bytes 2023-10-11T10:59:50.632 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1800 IOs, 635831296 bytes 2023-10-11T10:59:50.805 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1900 IOs, 665690112 bytes 2023-10-11T10:59:50.975 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2000 IOs, 700648448 bytes 2023-10-11T10:59:51.124 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2100 IOs, 731665408 bytes 2023-10-11T10:59:51.408 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2200 IOs, 820870144 bytes 2023-10-11T10:59:51.601 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2300 IOs, 867087360 bytes 2023-10-11T10:59:51.724 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2400 IOs, 880801792 bytes 2023-10-11T10:59:51.949 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2500 IOs, 936089600 bytes 2023-10-11T10:59:52.078 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2600 IOs, 950556672 bytes 2023-10-11T10:59:52.266 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2700 IOs, 991576064 bytes 2023-10-11T10:59:52.564 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2800 IOs, 1087595520 bytes 2023-10-11T10:59:52.676 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2900 IOs, 1092738048 bytes 2023-10-11T10:59:52.866 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3000 IOs, 1131205632 bytes 2023-10-11T10:59:52.968 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3100 IOs, 1139423232 bytes 2023-10-11T10:59:53.136 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3200 IOs, 1166395392 bytes 2023-10-11T10:59:53.359 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3300 IOs, 1218579456 bytes 2023-10-11T10:59:53.594 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3400 IOs, 1282033664 bytes 2023-10-11T10:59:53.844 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3500 IOs, 1378019328 bytes 2023-10-11T10:59:53.998 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3600 IOs, 1387722752 bytes 2023-10-11T10:59:54.169 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3700 IOs, 1411984384 bytes 2023-10-11T10:59:54.302 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3800 IOs, 1427436544 bytes 2023-10-11T10:59:54.467 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3900 IOs, 1459407872 bytes 2023-10-11T10:59:54.840 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4000 IOs, 1593330688 bytes 2023-10-11T10:59:55.173 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4100 IOs, 1708104704 bytes 2023-10-11T10:59:55.319 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4200 IOs, 1726291968 bytes 2023-10-11T10:59:55.511 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4300 IOs, 1768213504 bytes 2023-10-11T10:59:55.744 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4400 IOs, 1836531712 bytes 2023-10-11T10:59:55.928 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4500 IOs, 1896732672 bytes 2023-10-11T10:59:56.143 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4600 IOs, 1932626944 bytes 2023-10-11T10:59:56.308 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4700 IOs, 1957663744 bytes 2023-10-11T10:59:56.470 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4800 IOs, 1998394368 bytes 2023-10-11T10:59:56.478 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ error_code=124 2023-10-11T10:59:56.478 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ set -e 2023-10-11T10:59:56.478 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' 124 -eq 124 ']' 2023-10-11T10:59:56.478 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T10:59:56.479 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T10:59:56.479 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_36 2023-10-11T10:59:56.479 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ create_image_and_enable_mirror cluster2 mirror image_36 snapshot 128M 2023-10-11T10:59:56.479 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:59:56.479 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:59:56.480 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:59:56.480 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:59:56.480 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_36 2023-10-11T10:59:56.480 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:59:56.480 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local mode=snapshot 2023-10-11T10:59:56.480 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n snapshot ']' 2023-10-11T10:59:56.481 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:59:56.481 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ create_image cluster2 mirror image_36 128M 2023-10-11T10:59:56.481 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:59:56.481 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:59:56.481 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:59:56.482 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:59:56.482 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_36 2023-10-11T10:59:56.482 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:59:56.482 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local size=128 2023-10-11T10:59:56.482 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n 128M ']' 2023-10-11T10:59:56.482 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ size=128M 2023-10-11T10:59:56.483 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T10:59:56.483 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_36 2023-10-11T10:59:56.526 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' image = image ']' 2023-10-11T10:59:56.526 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ enable_mirror cluster2 mirror image_36 snapshot 2023-10-11T10:59:56.527 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:59:56.527 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:59:56.527 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_36 2023-10-11T10:59:56.527 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local mode=snapshot 2023-10-11T10:59:56.527 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_36 snapshot 2023-10-11T10:59:57.460 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:Mirroring enabled 2023-10-11T10:59:57.470 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 info mirror/image_36 2023-10-11T10:59:57.508 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:rbd image 'image_36': 2023-10-11T10:59:57.508 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: size 128 MiB in 32 objects 2023-10-11T10:59:57.508 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: order 22 (4 MiB objects) 2023-10-11T10:59:57.508 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: snapshot_count: 1 2023-10-11T10:59:57.508 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: id: 13944d68965d 2023-10-11T10:59:57.509 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: block_name_prefix: rbd_data.13944d68965d 2023-10-11T10:59:57.509 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: format: 2 2023-10-11T10:59:57.509 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-10-11T10:59:57.509 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: op_features: 2023-10-11T10:59:57.509 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: flags: 2023-10-11T10:59:57.510 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: create_timestamp: Wed Oct 11 10:59:56 2023 2023-10-11T10:59:57.510 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: access_timestamp: Wed Oct 11 10:59:56 2023 2023-10-11T10:59:57.510 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: modify_timestamp: Wed Oct 11 10:59:56 2023 2023-10-11T10:59:57.510 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring state: enabled 2023-10-11T10:59:57.510 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring mode: snapshot 2023-10-11T10:59:57.511 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring global id: 26586d31-8ccc-4ee9-b1da-f7f6be6b7fa9 2023-10-11T10:59:57.511 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring primary: true 2023-10-11T10:59:57.512 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n '' ']' 2023-10-11T10:59:57.512 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ stress_write_image cluster2 mirror image_36 2023-10-11T10:59:57.512 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T10:59:57.512 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T10:59:57.512 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_36 2023-10-11T10:59:57.513 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-10-11T10:59:57.513 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local duration=5 2023-10-11T10:59:57.513 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ set +e 2023-10-11T10:59:57.514 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_36 --debug-rbd=20 --debug-journaler=20 2023-10-11T10:59:57.708 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:100 IOs, 15233024 bytes 2023-10-11T10:59:57.905 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:200 IOs, 65454080 bytes 2023-10-11T10:59:58.013 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:300 IOs, 66342912 bytes 2023-10-11T10:59:58.219 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:400 IOs, 121330688 bytes 2023-10-11T10:59:58.396 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:500 IOs, 152382464 bytes 2023-10-11T10:59:58.600 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:600 IOs, 208055296 bytes 2023-10-11T10:59:58.715 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:700 IOs, 208674816 bytes 2023-10-11T10:59:58.953 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:800 IOs, 275280896 bytes 2023-10-11T10:59:59.081 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:900 IOs, 306138112 bytes 2023-10-11T10:59:59.248 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1000 IOs, 311547904 bytes 2023-10-11T10:59:59.464 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1100 IOs, 378490880 bytes 2023-10-11T10:59:59.642 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1200 IOs, 401397760 bytes 2023-10-11T10:59:59.856 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1300 IOs, 445173760 bytes 2023-10-11T11:00:00.030 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1400 IOs, 478255104 bytes 2023-10-11T11:00:00.199 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1500 IOs, 503188480 bytes 2023-10-11T11:00:00.350 INFO:tasks.ceph.cluster2.osd.1.smithi111.stderr:2023-10-11T11:00:00.347+0000 7f9628714700 -1 reset not still connected to 0x564284229e10 2023-10-11T11:00:00.409 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1600 IOs, 558092288 bytes 2023-10-11T11:00:00.644 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1700 IOs, 620351488 bytes 2023-10-11T11:00:00.930 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1800 IOs, 706124800 bytes 2023-10-11T11:00:01.189 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1900 IOs, 783771648 bytes 2023-10-11T11:00:01.356 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2000 IOs, 815602688 bytes 2023-10-11T11:00:01.559 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2100 IOs, 863496192 bytes 2023-10-11T11:00:01.753 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2200 IOs, 907866112 bytes 2023-10-11T11:00:01.941 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2300 IOs, 953950208 bytes 2023-10-11T11:00:02.112 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2400 IOs, 985617408 bytes 2023-10-11T11:00:02.297 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2500 IOs, 1028123648 bytes 2023-10-11T11:00:02.428 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2600 IOs, 1044737024 bytes 2023-10-11T11:00:02.522 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ error_code=124 2023-10-11T11:00:02.522 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ set -e 2023-10-11T11:00:02.522 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' 124 -eq 124 ']' 2023-10-11T11:00:02.523 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:00:02.523 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T11:00:02.523 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_37 2023-10-11T11:00:02.523 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ create_image_and_enable_mirror cluster2 mirror image_37 snapshot 128M 2023-10-11T11:00:02.523 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:00:02.523 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T11:00:02.524 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:00:02.524 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T11:00:02.524 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_37 2023-10-11T11:00:02.524 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T11:00:02.524 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local mode=snapshot 2023-10-11T11:00:02.524 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n snapshot ']' 2023-10-11T11:00:02.525 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T11:00:02.525 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ create_image cluster2 mirror image_37 128M 2023-10-11T11:00:02.525 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:00:02.525 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T11:00:02.525 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:00:02.526 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T11:00:02.526 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_37 2023-10-11T11:00:02.526 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T11:00:02.526 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local size=128 2023-10-11T11:00:02.526 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n 128M ']' 2023-10-11T11:00:02.526 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ size=128M 2023-10-11T11:00:02.527 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T11:00:02.527 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_37 2023-10-11T11:00:02.571 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' image = image ']' 2023-10-11T11:00:02.571 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ enable_mirror cluster2 mirror image_37 snapshot 2023-10-11T11:00:02.572 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:00:02.572 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:00:02.572 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_37 2023-10-11T11:00:02.572 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local mode=snapshot 2023-10-11T11:00:02.572 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_37 snapshot 2023-10-11T11:00:03.416 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:Mirroring enabled 2023-10-11T11:00:03.427 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 info mirror/image_37 2023-10-11T11:00:03.464 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:rbd image 'image_37': 2023-10-11T11:00:03.465 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: size 128 MiB in 32 objects 2023-10-11T11:00:03.465 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: order 22 (4 MiB objects) 2023-10-11T11:00:03.465 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: snapshot_count: 1 2023-10-11T11:00:03.465 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: id: 139d13732fd0 2023-10-11T11:00:03.465 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: block_name_prefix: rbd_data.139d13732fd0 2023-10-11T11:00:03.466 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: format: 2 2023-10-11T11:00:03.466 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-10-11T11:00:03.466 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: op_features: 2023-10-11T11:00:03.466 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: flags: 2023-10-11T11:00:03.466 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: create_timestamp: Wed Oct 11 11:00:02 2023 2023-10-11T11:00:03.467 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: access_timestamp: Wed Oct 11 11:00:02 2023 2023-10-11T11:00:03.467 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: modify_timestamp: Wed Oct 11 11:00:02 2023 2023-10-11T11:00:03.467 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring state: enabled 2023-10-11T11:00:03.467 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring mode: snapshot 2023-10-11T11:00:03.468 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring global id: 71f091e9-defc-4548-9e77-73505b063ba4 2023-10-11T11:00:03.468 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring primary: true 2023-10-11T11:00:03.469 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n '' ']' 2023-10-11T11:00:03.469 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ stress_write_image cluster2 mirror image_37 2023-10-11T11:00:03.469 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:00:03.469 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:00:03.469 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_37 2023-10-11T11:00:03.470 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-10-11T11:00:03.470 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local duration=5 2023-10-11T11:00:03.471 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ set +e 2023-10-11T11:00:03.471 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_37 --debug-rbd=20 --debug-journaler=20 2023-10-11T11:00:03.666 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:100 IOs, 3489792 bytes 2023-10-11T11:00:03.857 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:200 IOs, 48087040 bytes 2023-10-11T11:00:04.030 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:300 IOs, 83743744 bytes 2023-10-11T11:00:04.168 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:400 IOs, 97631232 bytes 2023-10-11T11:00:04.384 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:500 IOs, 148858880 bytes 2023-10-11T11:00:04.695 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:600 IOs, 250908672 bytes 2023-10-11T11:00:04.876 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:700 IOs, 283390976 bytes 2023-10-11T11:00:05.082 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:800 IOs, 330661888 bytes 2023-10-11T11:00:05.211 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:900 IOs, 361970688 bytes 2023-10-11T11:00:05.449 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1000 IOs, 391685120 bytes 2023-10-11T11:00:05.602 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1100 IOs, 429427712 bytes 2023-10-11T11:00:05.865 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1200 IOs, 492973056 bytes 2023-10-11T11:00:06.069 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1300 IOs, 536300544 bytes 2023-10-11T11:00:06.273 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1400 IOs, 589125632 bytes 2023-10-11T11:00:06.420 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1500 IOs, 610718720 bytes 2023-10-11T11:00:06.624 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1600 IOs, 660559872 bytes 2023-10-11T11:00:06.806 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1700 IOs, 703609856 bytes 2023-10-11T11:00:07.008 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1800 IOs, 747408384 bytes 2023-10-11T11:00:07.182 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1900 IOs, 786900992 bytes 2023-10-11T11:00:07.377 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2000 IOs, 823251968 bytes 2023-10-11T11:00:07.484 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2100 IOs, 823854080 bytes 2023-10-11T11:00:07.592 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2200 IOs, 828038144 bytes 2023-10-11T11:00:07.726 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2300 IOs, 841914368 bytes 2023-10-11T11:00:07.948 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2400 IOs, 893775872 bytes 2023-10-11T11:00:08.116 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2500 IOs, 922769408 bytes 2023-10-11T11:00:08.242 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2600 IOs, 929531904 bytes 2023-10-11T11:00:08.360 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2700 IOs, 955921408 bytes 2023-10-11T11:00:08.475 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ error_code=124 2023-10-11T11:00:08.475 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ set -e 2023-10-11T11:00:08.475 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' 124 -eq 124 ']' 2023-10-11T11:00:08.475 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:00:08.475 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T11:00:08.476 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_38 2023-10-11T11:00:08.476 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ create_image_and_enable_mirror cluster2 mirror image_38 snapshot 128M 2023-10-11T11:00:08.476 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:00:08.476 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T11:00:08.476 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:00:08.476 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T11:00:08.477 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_38 2023-10-11T11:00:08.477 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T11:00:08.477 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local mode=snapshot 2023-10-11T11:00:08.477 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n snapshot ']' 2023-10-11T11:00:08.477 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T11:00:08.477 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ create_image cluster2 mirror image_38 128M 2023-10-11T11:00:08.478 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:00:08.478 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T11:00:08.478 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:00:08.478 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T11:00:08.478 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_38 2023-10-11T11:00:08.478 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T11:00:08.479 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local size=128 2023-10-11T11:00:08.479 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n 128M ']' 2023-10-11T11:00:08.479 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ size=128M 2023-10-11T11:00:08.479 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T11:00:08.479 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_38 2023-10-11T11:00:08.523 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' image = image ']' 2023-10-11T11:00:08.524 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ enable_mirror cluster2 mirror image_38 snapshot 2023-10-11T11:00:08.524 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:00:08.524 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:00:08.524 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_38 2023-10-11T11:00:08.524 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local mode=snapshot 2023-10-11T11:00:08.524 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_38 snapshot 2023-10-11T11:00:09.452 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:Mirroring enabled 2023-10-11T11:00:09.463 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 info mirror/image_38 2023-10-11T11:00:09.500 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:rbd image 'image_38': 2023-10-11T11:00:09.500 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: size 128 MiB in 32 objects 2023-10-11T11:00:09.500 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: order 22 (4 MiB objects) 2023-10-11T11:00:09.500 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: snapshot_count: 1 2023-10-11T11:00:09.501 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: id: 13a67a4d073e 2023-10-11T11:00:09.501 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: block_name_prefix: rbd_data.13a67a4d073e 2023-10-11T11:00:09.501 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: format: 2 2023-10-11T11:00:09.501 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-10-11T11:00:09.501 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: op_features: 2023-10-11T11:00:09.501 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: flags: 2023-10-11T11:00:09.502 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: create_timestamp: Wed Oct 11 11:00:08 2023 2023-10-11T11:00:09.502 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: access_timestamp: Wed Oct 11 11:00:08 2023 2023-10-11T11:00:09.502 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: modify_timestamp: Wed Oct 11 11:00:08 2023 2023-10-11T11:00:09.502 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring state: enabled 2023-10-11T11:00:09.502 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring mode: snapshot 2023-10-11T11:00:09.502 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring global id: 8f0a6f87-4fd6-4ab1-8d9b-69b67766f0e3 2023-10-11T11:00:09.503 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring primary: true 2023-10-11T11:00:09.503 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n '' ']' 2023-10-11T11:00:09.504 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ stress_write_image cluster2 mirror image_38 2023-10-11T11:00:09.504 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:00:09.504 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:00:09.504 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_38 2023-10-11T11:00:09.504 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-10-11T11:00:09.505 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local duration=12 2023-10-11T11:00:09.505 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ set +e 2023-10-11T11:00:09.505 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_38 --debug-rbd=20 --debug-journaler=20 2023-10-11T11:00:09.767 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:100 IOs, 44254208 bytes 2023-10-11T11:00:09.979 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:200 IOs, 103254016 bytes 2023-10-11T11:00:10.117 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:300 IOs, 125427712 bytes 2023-10-11T11:00:10.343 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:400 IOs, 180044800 bytes 2023-10-11T11:00:10.563 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:500 IOs, 247933952 bytes 2023-10-11T11:00:10.742 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:600 IOs, 282878976 bytes 2023-10-11T11:00:11.035 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:700 IOs, 380964864 bytes 2023-10-11T11:00:11.208 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:800 IOs, 415049728 bytes 2023-10-11T11:00:11.419 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:900 IOs, 470789120 bytes 2023-10-11T11:00:11.536 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1000 IOs, 473378816 bytes 2023-10-11T11:00:11.743 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1100 IOs, 525450240 bytes 2023-10-11T11:00:11.891 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1200 IOs, 540471296 bytes 2023-10-11T11:00:12.183 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1300 IOs, 593676288 bytes 2023-10-11T11:00:12.336 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1400 IOs, 616579072 bytes 2023-10-11T11:00:12.518 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1500 IOs, 665214976 bytes 2023-10-11T11:00:12.691 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1600 IOs, 695712768 bytes 2023-10-11T11:00:12.904 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1700 IOs, 746702848 bytes 2023-10-11T11:00:13.070 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1800 IOs, 781243392 bytes 2023-10-11T11:00:13.257 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1900 IOs, 821184512 bytes 2023-10-11T11:00:13.543 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2000 IOs, 913207296 bytes 2023-10-11T11:00:13.671 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2100 IOs, 928259072 bytes 2023-10-11T11:00:13.869 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2200 IOs, 975680512 bytes 2023-10-11T11:00:14.076 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2300 IOs, 1032361984 bytes 2023-10-11T11:00:14.220 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2400 IOs, 1053152256 bytes 2023-10-11T11:00:14.419 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2500 IOs, 1109160960 bytes 2023-10-11T11:00:14.578 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2600 IOs, 1141042176 bytes 2023-10-11T11:00:14.714 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2700 IOs, 1162413056 bytes 2023-10-11T11:00:14.846 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2800 IOs, 1167875072 bytes 2023-10-11T11:00:15.052 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2900 IOs, 1216918528 bytes 2023-10-11T11:00:15.198 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3000 IOs, 1240601600 bytes 2023-10-11T11:00:15.468 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3100 IOs, 1327589376 bytes 2023-10-11T11:00:15.584 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3200 IOs, 1328199680 bytes 2023-10-11T11:00:15.707 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3300 IOs, 1340319744 bytes 2023-10-11T11:00:15.840 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3400 IOs, 1353784320 bytes 2023-10-11T11:00:16.026 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3500 IOs, 1394605056 bytes 2023-10-11T11:00:16.223 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3600 IOs, 1441666048 bytes 2023-10-11T11:00:16.399 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3700 IOs, 1474637824 bytes 2023-10-11T11:00:16.631 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3800 IOs, 1538800640 bytes 2023-10-11T11:00:16.788 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3900 IOs, 1562992640 bytes 2023-10-11T11:00:16.951 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4000 IOs, 1593510912 bytes 2023-10-11T11:00:17.106 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4100 IOs, 1618348032 bytes 2023-10-11T11:00:17.298 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4200 IOs, 1671660544 bytes 2023-10-11T11:00:17.536 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4300 IOs, 1715001344 bytes 2023-10-11T11:00:17.662 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4400 IOs, 1726216192 bytes 2023-10-11T11:00:17.873 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4500 IOs, 1773648896 bytes 2023-10-11T11:00:18.038 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4600 IOs, 1802295296 bytes 2023-10-11T11:00:18.270 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4700 IOs, 1877614592 bytes 2023-10-11T11:00:18.502 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4800 IOs, 1930863616 bytes 2023-10-11T11:00:18.770 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4900 IOs, 2013523968 bytes 2023-10-11T11:00:18.921 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5000 IOs, 2040213504 bytes 2023-10-11T11:00:19.107 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5100 IOs, 2086781952 bytes 2023-10-11T11:00:19.236 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5200 IOs, 2096120832 bytes 2023-10-11T11:00:19.519 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5300 IOs, 2187612160 bytes 2023-10-11T11:00:19.772 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5400 IOs, 2254807040 bytes 2023-10-11T11:00:19.943 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5500 IOs, 2289979392 bytes 2023-10-11T11:00:20.130 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5600 IOs, 2334007296 bytes 2023-10-11T11:00:20.369 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5700 IOs, 2405090304 bytes 2023-10-11T11:00:20.567 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5800 IOs, 2467104768 bytes 2023-10-11T11:00:20.728 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5900 IOs, 2501779456 bytes 2023-10-11T11:00:20.970 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6000 IOs, 2547050496 bytes 2023-10-11T11:00:21.168 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6100 IOs, 2591817728 bytes 2023-10-11T11:00:21.313 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6200 IOs, 2611399680 bytes 2023-10-11T11:00:21.426 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6300 IOs, 2619107328 bytes 2023-10-11T11:00:21.512 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ error_code=124 2023-10-11T11:00:21.512 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ set -e 2023-10-11T11:00:21.512 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' 124 -eq 124 ']' 2023-10-11T11:00:21.512 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:00:21.513 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T11:00:21.513 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_39 2023-10-11T11:00:21.513 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ create_image_and_enable_mirror cluster2 mirror image_39 snapshot 128M 2023-10-11T11:00:21.513 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:00:21.513 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T11:00:21.513 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:00:21.514 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T11:00:21.514 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_39 2023-10-11T11:00:21.514 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T11:00:21.514 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local mode=snapshot 2023-10-11T11:00:21.514 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n snapshot ']' 2023-10-11T11:00:21.515 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T11:00:21.515 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ create_image cluster2 mirror image_39 128M 2023-10-11T11:00:21.515 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:00:21.515 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T11:00:21.515 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:00:21.515 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T11:00:21.516 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_39 2023-10-11T11:00:21.516 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T11:00:21.516 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local size=128 2023-10-11T11:00:21.516 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n 128M ']' 2023-10-11T11:00:21.516 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ size=128M 2023-10-11T11:00:21.516 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T11:00:21.517 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_39 2023-10-11T11:00:21.562 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' image = image ']' 2023-10-11T11:00:21.562 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ enable_mirror cluster2 mirror image_39 snapshot 2023-10-11T11:00:21.562 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:00:21.562 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:00:21.562 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_39 2023-10-11T11:00:21.563 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local mode=snapshot 2023-10-11T11:00:21.563 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_39 snapshot 2023-10-11T11:00:22.567 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:Mirroring enabled 2023-10-11T11:00:22.578 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 info mirror/image_39 2023-10-11T11:00:22.616 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:rbd image 'image_39': 2023-10-11T11:00:22.616 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: size 128 MiB in 32 objects 2023-10-11T11:00:22.616 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: order 22 (4 MiB objects) 2023-10-11T11:00:22.616 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: snapshot_count: 1 2023-10-11T11:00:22.617 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: id: 13af13b5a346 2023-10-11T11:00:22.617 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: block_name_prefix: rbd_data.13af13b5a346 2023-10-11T11:00:22.617 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: format: 2 2023-10-11T11:00:22.617 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-10-11T11:00:22.617 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: op_features: 2023-10-11T11:00:22.617 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: flags: 2023-10-11T11:00:22.618 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: create_timestamp: Wed Oct 11 11:00:21 2023 2023-10-11T11:00:22.618 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: access_timestamp: Wed Oct 11 11:00:21 2023 2023-10-11T11:00:22.618 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: modify_timestamp: Wed Oct 11 11:00:21 2023 2023-10-11T11:00:22.618 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring state: enabled 2023-10-11T11:00:22.618 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring mode: snapshot 2023-10-11T11:00:22.618 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring global id: 20fcc198-7ca6-4408-82dd-c95c583a202d 2023-10-11T11:00:22.619 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring primary: true 2023-10-11T11:00:22.619 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n '' ']' 2023-10-11T11:00:22.620 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ stress_write_image cluster2 mirror image_39 2023-10-11T11:00:22.620 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:00:22.620 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:00:22.620 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_39 2023-10-11T11:00:22.620 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-10-11T11:00:22.621 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local duration=6 2023-10-11T11:00:22.621 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ set +e 2023-10-11T11:00:22.621 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_39 --debug-rbd=20 --debug-journaler=20 2023-10-11T11:00:22.800 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:100 IOs, 614400 bytes 2023-10-11T11:00:22.977 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:200 IOs, 32650240 bytes 2023-10-11T11:00:23.119 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:300 IOs, 54189056 bytes 2023-10-11T11:00:23.355 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:400 IOs, 129331200 bytes 2023-10-11T11:00:23.493 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:500 IOs, 136629248 bytes 2023-10-11T11:00:23.664 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:600 IOs, 175057920 bytes 2023-10-11T11:00:23.877 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:700 IOs, 226929664 bytes 2023-10-11T11:00:24.043 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:800 IOs, 258885632 bytes 2023-10-11T11:00:24.194 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:900 IOs, 273922048 bytes 2023-10-11T11:00:24.508 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1000 IOs, 345660416 bytes 2023-10-11T11:00:24.640 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1100 IOs, 369530880 bytes 2023-10-11T11:00:24.863 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1200 IOs, 411126784 bytes 2023-10-11T11:00:24.967 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1300 IOs, 411743232 bytes 2023-10-11T11:00:25.199 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1400 IOs, 478213120 bytes 2023-10-11T11:00:25.377 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1500 IOs, 513712128 bytes 2023-10-11T11:00:25.561 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1600 IOs, 568734720 bytes 2023-10-11T11:00:25.798 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1700 IOs, 620411904 bytes 2023-10-11T11:00:25.968 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1800 IOs, 651152384 bytes 2023-10-11T11:00:26.159 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1900 IOs, 693989376 bytes 2023-10-11T11:00:26.387 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2000 IOs, 770212864 bytes 2023-10-11T11:00:26.526 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2100 IOs, 778277888 bytes 2023-10-11T11:00:26.548 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.29706634714663116) > weight (0.25) 2023-10-11T11:00:26.548 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.38671279850222895) > weight (0.25) 2023-10-11T11:00:26.548 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.9620405903493526) > weight (0.25) 2023-10-11T11:00:26.549 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.3 2023-10-11T11:00:26.549 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Sent signal 15 2023-10-11T11:00:26.549 INFO:tasks.rbd_mirror_thrash:waiting for 3 secs before reviving daemons 2023-10-11T11:00:26.550 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi111.stderr:2023-10-11T11:00:26.549+0000 7f7cf9f36700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 (PID: 116087) UID: 1000 2023-10-11T11:00:26.757 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2200 IOs, 845108224 bytes 2023-10-11T11:00:26.975 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2300 IOs, 884103168 bytes 2023-10-11T11:00:27.174 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2400 IOs, 934021120 bytes 2023-10-11T11:00:27.334 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2500 IOs, 962944000 bytes 2023-10-11T11:00:27.557 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2600 IOs, 1023638528 bytes 2023-10-11T11:00:27.680 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2700 IOs, 1029480448 bytes 2023-10-11T11:00:27.847 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2800 IOs, 1064312832 bytes 2023-10-11T11:00:28.048 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2900 IOs, 1107951616 bytes 2023-10-11T11:00:28.172 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3000 IOs, 1120310272 bytes 2023-10-11T11:00:28.388 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3100 IOs, 1177570304 bytes 2023-10-11T11:00:28.630 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ error_code=124 2023-10-11T11:00:28.630 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ set -e 2023-10-11T11:00:28.630 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' 124 -eq 124 ']' 2023-10-11T11:00:28.630 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:00:28.631 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T11:00:28.631 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_40 2023-10-11T11:00:28.631 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ create_image_and_enable_mirror cluster2 mirror image_40 snapshot 128M 2023-10-11T11:00:28.631 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:00:28.631 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T11:00:28.632 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:00:28.632 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T11:00:28.632 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_40 2023-10-11T11:00:28.632 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T11:00:28.632 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local mode=snapshot 2023-10-11T11:00:28.632 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n snapshot ']' 2023-10-11T11:00:28.633 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T11:00:28.633 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ create_image cluster2 mirror image_40 128M 2023-10-11T11:00:28.633 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:00:28.633 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T11:00:28.633 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:00:28.634 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T11:00:28.634 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_40 2023-10-11T11:00:28.634 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T11:00:28.634 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local size=128 2023-10-11T11:00:28.634 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n 128M ']' 2023-10-11T11:00:28.634 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ size=128M 2023-10-11T11:00:28.635 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T11:00:28.635 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_40 2023-10-11T11:00:28.710 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' image = image ']' 2023-10-11T11:00:28.710 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ enable_mirror cluster2 mirror image_40 snapshot 2023-10-11T11:00:28.710 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:00:28.710 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:00:28.710 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_40 2023-10-11T11:00:28.711 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local mode=snapshot 2023-10-11T11:00:28.711 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_40 snapshot 2023-10-11T11:00:29.550 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.3 2023-10-11T11:00:29.550 INFO:teuthology.orchestra.run:waiting for 600 2023-10-11T11:00:29.550 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.3 2023-10-11T11:00:29.551 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Restarting daemon 2023-10-11T11:00:29.551 DEBUG:teuthology.orchestra.run.smithi111:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 2023-10-11T11:00:29.554 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Started 2023-10-11T11:00:29.554 INFO:tasks.rbd_mirror_thrash:waiting for 64 secs before thrashing 2023-10-11T11:00:29.594 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:Mirroring enabled 2023-10-11T11:00:29.605 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 info mirror/image_40 2023-10-11T11:00:29.643 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:rbd image 'image_40': 2023-10-11T11:00:29.644 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: size 128 MiB in 32 objects 2023-10-11T11:00:29.644 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: order 22 (4 MiB objects) 2023-10-11T11:00:29.644 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: snapshot_count: 1 2023-10-11T11:00:29.644 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: id: 13b77c87ef5e 2023-10-11T11:00:29.644 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: block_name_prefix: rbd_data.13b77c87ef5e 2023-10-11T11:00:29.645 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: format: 2 2023-10-11T11:00:29.645 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-10-11T11:00:29.645 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: op_features: 2023-10-11T11:00:29.645 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: flags: 2023-10-11T11:00:29.645 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: create_timestamp: Wed Oct 11 11:00:28 2023 2023-10-11T11:00:29.645 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: access_timestamp: Wed Oct 11 11:00:28 2023 2023-10-11T11:00:29.646 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: modify_timestamp: Wed Oct 11 11:00:28 2023 2023-10-11T11:00:29.646 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring state: enabled 2023-10-11T11:00:29.646 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring mode: snapshot 2023-10-11T11:00:29.646 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring global id: 932fa6e9-efac-4d1a-a4b3-2f155e3296d4 2023-10-11T11:00:29.647 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring primary: true 2023-10-11T11:00:29.647 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n '' ']' 2023-10-11T11:00:29.648 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ stress_write_image cluster2 mirror image_40 2023-10-11T11:00:29.648 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:00:29.648 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:00:29.648 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_40 2023-10-11T11:00:29.649 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-10-11T11:00:29.649 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local duration=9 2023-10-11T11:00:29.649 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ set +e 2023-10-11T11:00:29.650 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_40 --debug-rbd=20 --debug-journaler=20 2023-10-11T11:00:30.117 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:100 IOs, 49390592 bytes 2023-10-11T11:00:30.283 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:200 IOs, 85238784 bytes 2023-10-11T11:00:30.458 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:300 IOs, 112522240 bytes 2023-10-11T11:00:30.711 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:400 IOs, 192929792 bytes 2023-10-11T11:00:30.896 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:500 IOs, 245597184 bytes 2023-10-11T11:00:31.097 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:600 IOs, 282692608 bytes 2023-10-11T11:00:31.331 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:700 IOs, 342874112 bytes 2023-10-11T11:00:31.509 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:800 IOs, 385369088 bytes 2023-10-11T11:00:31.877 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:900 IOs, 423576576 bytes 2023-10-11T11:00:32.030 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1000 IOs, 447049728 bytes 2023-10-11T11:00:32.263 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1100 IOs, 508244992 bytes 2023-10-11T11:00:32.474 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1200 IOs, 570411008 bytes 2023-10-11T11:00:32.765 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1300 IOs, 650886144 bytes 2023-10-11T11:00:32.918 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1400 IOs, 673486848 bytes 2023-10-11T11:00:33.276 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1500 IOs, 807862272 bytes 2023-10-11T11:00:33.428 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1600 IOs, 828937216 bytes 2023-10-11T11:00:33.722 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1700 IOs, 920142848 bytes 2023-10-11T11:00:33.959 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1800 IOs, 980899840 bytes 2023-10-11T11:00:34.161 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1900 IOs, 1025417216 bytes 2023-10-11T11:00:34.359 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2000 IOs, 1067966464 bytes 2023-10-11T11:00:34.556 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2100 IOs, 1107551232 bytes 2023-10-11T11:00:34.756 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2200 IOs, 1155374080 bytes 2023-10-11T11:00:34.949 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2300 IOs, 1196570624 bytes 2023-10-11T11:00:35.297 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2400 IOs, 1319360512 bytes 2023-10-11T11:00:35.466 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2500 IOs, 1350507520 bytes 2023-10-11T11:00:35.590 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2600 IOs, 1360134144 bytes 2023-10-11T11:00:35.744 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2700 IOs, 1381986304 bytes 2023-10-11T11:00:35.887 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2800 IOs, 1395901440 bytes 2023-10-11T11:00:36.052 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2900 IOs, 1429505024 bytes 2023-10-11T11:00:36.256 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3000 IOs, 1479283712 bytes 2023-10-11T11:00:36.400 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3100 IOs, 1494703104 bytes 2023-10-11T11:00:36.547 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3200 IOs, 1516708864 bytes 2023-10-11T11:00:36.732 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3300 IOs, 1571362816 bytes 2023-10-11T11:00:37.026 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3400 IOs, 1646149632 bytes 2023-10-11T11:00:37.155 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3500 IOs, 1657401344 bytes 2023-10-11T11:00:37.310 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3600 IOs, 1682954240 bytes 2023-10-11T11:00:37.573 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3700 IOs, 1760815104 bytes 2023-10-11T11:00:37.779 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3800 IOs, 1829860352 bytes 2023-10-11T11:00:37.977 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3900 IOs, 1848659968 bytes 2023-10-11T11:00:38.129 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4000 IOs, 1872592896 bytes 2023-10-11T11:00:38.268 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4100 IOs, 1883420672 bytes 2023-10-11T11:00:38.417 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4200 IOs, 1906473984 bytes 2023-10-11T11:00:38.635 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4300 IOs, 1986472960 bytes 2023-10-11T11:00:38.657 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ error_code=124 2023-10-11T11:00:38.658 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ set -e 2023-10-11T11:00:38.658 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' 124 -eq 124 ']' 2023-10-11T11:00:38.658 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:00:38.658 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T11:00:38.658 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_41 2023-10-11T11:00:38.658 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ create_image_and_enable_mirror cluster2 mirror image_41 snapshot 128M 2023-10-11T11:00:38.659 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:00:38.659 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T11:00:38.659 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:00:38.659 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T11:00:38.659 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_41 2023-10-11T11:00:38.659 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T11:00:38.660 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local mode=snapshot 2023-10-11T11:00:38.660 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n snapshot ']' 2023-10-11T11:00:38.660 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T11:00:38.660 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ create_image cluster2 mirror image_41 128M 2023-10-11T11:00:38.661 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:00:38.661 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T11:00:38.661 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:00:38.661 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T11:00:38.661 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_41 2023-10-11T11:00:38.661 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T11:00:38.662 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local size=128 2023-10-11T11:00:38.662 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n 128M ']' 2023-10-11T11:00:38.662 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ size=128M 2023-10-11T11:00:38.662 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T11:00:38.662 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_41 2023-10-11T11:00:38.706 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' image = image ']' 2023-10-11T11:00:38.706 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ enable_mirror cluster2 mirror image_41 snapshot 2023-10-11T11:00:38.706 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:00:38.707 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:00:38.707 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_41 2023-10-11T11:00:38.707 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local mode=snapshot 2023-10-11T11:00:38.707 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_41 snapshot 2023-10-11T11:00:39.650 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:Mirroring enabled 2023-10-11T11:00:39.661 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 info mirror/image_41 2023-10-11T11:00:39.699 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:rbd image 'image_41': 2023-10-11T11:00:39.699 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: size 128 MiB in 32 objects 2023-10-11T11:00:39.699 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: order 22 (4 MiB objects) 2023-10-11T11:00:39.699 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: snapshot_count: 1 2023-10-11T11:00:39.699 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: id: 13ccf5539785 2023-10-11T11:00:39.700 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: block_name_prefix: rbd_data.13ccf5539785 2023-10-11T11:00:39.700 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: format: 2 2023-10-11T11:00:39.700 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-10-11T11:00:39.700 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: op_features: 2023-10-11T11:00:39.700 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: flags: 2023-10-11T11:00:39.700 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: create_timestamp: Wed Oct 11 11:00:38 2023 2023-10-11T11:00:39.701 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: access_timestamp: Wed Oct 11 11:00:38 2023 2023-10-11T11:00:39.701 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: modify_timestamp: Wed Oct 11 11:00:38 2023 2023-10-11T11:00:39.701 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring state: enabled 2023-10-11T11:00:39.701 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring mode: snapshot 2023-10-11T11:00:39.701 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring global id: 981fef91-675f-4aae-8d59-cba97f3e8943 2023-10-11T11:00:39.702 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring primary: true 2023-10-11T11:00:39.702 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n '' ']' 2023-10-11T11:00:39.703 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ stress_write_image cluster2 mirror image_41 2023-10-11T11:00:39.703 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:00:39.703 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:00:39.703 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_41 2023-10-11T11:00:39.703 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-10-11T11:00:39.704 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local duration=5 2023-10-11T11:00:39.704 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ set +e 2023-10-11T11:00:39.704 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_41 --debug-rbd=20 --debug-journaler=20 2023-10-11T11:00:40.034 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:100 IOs, 80957440 bytes 2023-10-11T11:00:40.289 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:200 IOs, 163948544 bytes 2023-10-11T11:00:40.481 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:300 IOs, 202864640 bytes 2023-10-11T11:00:40.693 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:400 IOs, 256624640 bytes 2023-10-11T11:00:40.805 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:500 IOs, 257228800 bytes 2023-10-11T11:00:40.961 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:600 IOs, 293601280 bytes 2023-10-11T11:00:41.262 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:700 IOs, 371405824 bytes 2023-10-11T11:00:41.388 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:800 IOs, 383594496 bytes 2023-10-11T11:00:41.620 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:900 IOs, 433937408 bytes 2023-10-11T11:00:41.784 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1000 IOs, 478519296 bytes 2023-10-11T11:00:42.014 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1100 IOs, 521565184 bytes 2023-10-11T11:00:42.237 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1200 IOs, 567222272 bytes 2023-10-11T11:00:42.412 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1300 IOs, 599385088 bytes 2023-10-11T11:00:42.560 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2023-10-11T11:00:42.560 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2023-10-11T11:00:42.560 INFO:tasks.rbd_mirror_thrash:waiting for 3 secs before reviving daemons 2023-10-11T11:00:42.561 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi111.stderr:2023-10-11T11:00:42.560+0000 7f0a6b76e700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 117551) UID: 1000 2023-10-11T11:00:42.648 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1400 IOs, 663170048 bytes 2023-10-11T11:00:42.897 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1500 IOs, 715563008 bytes 2023-10-11T11:00:43.159 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1600 IOs, 783703040 bytes 2023-10-11T11:00:43.430 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1700 IOs, 872177664 bytes 2023-10-11T11:00:43.610 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1800 IOs, 917573632 bytes 2023-10-11T11:00:43.768 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1900 IOs, 948461568 bytes 2023-10-11T11:00:43.952 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2000 IOs, 992264192 bytes 2023-10-11T11:00:44.208 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2100 IOs, 1082543104 bytes 2023-10-11T11:00:44.379 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2200 IOs, 1110298624 bytes 2023-10-11T11:00:44.494 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2300 IOs, 1114649600 bytes 2023-10-11T11:00:44.604 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2400 IOs, 1123406848 bytes 2023-10-11T11:00:44.716 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ error_code=124 2023-10-11T11:00:44.716 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ set -e 2023-10-11T11:00:44.717 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' 124 -eq 124 ']' 2023-10-11T11:00:44.717 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:00:44.717 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T11:00:44.717 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_42 2023-10-11T11:00:44.717 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ create_image_and_enable_mirror cluster2 mirror image_42 snapshot 128M 2023-10-11T11:00:44.718 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:00:44.718 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T11:00:44.718 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:00:44.718 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T11:00:44.718 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_42 2023-10-11T11:00:44.718 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T11:00:44.719 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local mode=snapshot 2023-10-11T11:00:44.719 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n snapshot ']' 2023-10-11T11:00:44.719 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T11:00:44.719 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ create_image cluster2 mirror image_42 128M 2023-10-11T11:00:44.719 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:00:44.720 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T11:00:44.720 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:00:44.720 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T11:00:44.720 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_42 2023-10-11T11:00:44.720 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T11:00:44.721 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local size=128 2023-10-11T11:00:44.721 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n 128M ']' 2023-10-11T11:00:44.721 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ size=128M 2023-10-11T11:00:44.721 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T11:00:44.721 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_42 2023-10-11T11:00:44.767 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' image = image ']' 2023-10-11T11:00:44.767 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ enable_mirror cluster2 mirror image_42 snapshot 2023-10-11T11:00:44.768 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:00:44.768 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:00:44.768 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_42 2023-10-11T11:00:44.768 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local mode=snapshot 2023-10-11T11:00:44.768 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_42 snapshot 2023-10-11T11:00:45.561 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2023-10-11T11:00:45.562 INFO:teuthology.orchestra.run:waiting for 600 2023-10-11T11:00:45.562 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2023-10-11T11:00:45.562 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2023-10-11T11:00:45.562 DEBUG:teuthology.orchestra.run.smithi111:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2023-10-11T11:00:45.565 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2023-10-11T11:00:45.565 INFO:tasks.rbd_mirror_thrash:waiting for 84 secs before thrashing 2023-10-11T11:00:45.667 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:Mirroring enabled 2023-10-11T11:00:45.678 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 info mirror/image_42 2023-10-11T11:00:45.716 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:rbd image 'image_42': 2023-10-11T11:00:45.716 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: size 128 MiB in 32 objects 2023-10-11T11:00:45.716 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: order 22 (4 MiB objects) 2023-10-11T11:00:45.716 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: snapshot_count: 1 2023-10-11T11:00:45.716 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: id: 13d7546455b3 2023-10-11T11:00:45.717 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: block_name_prefix: rbd_data.13d7546455b3 2023-10-11T11:00:45.717 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: format: 2 2023-10-11T11:00:45.717 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-10-11T11:00:45.717 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: op_features: 2023-10-11T11:00:45.717 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: flags: 2023-10-11T11:00:45.718 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: create_timestamp: Wed Oct 11 11:00:44 2023 2023-10-11T11:00:45.718 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: access_timestamp: Wed Oct 11 11:00:44 2023 2023-10-11T11:00:45.718 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: modify_timestamp: Wed Oct 11 11:00:44 2023 2023-10-11T11:00:45.718 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring state: enabled 2023-10-11T11:00:45.718 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring mode: snapshot 2023-10-11T11:00:45.719 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring global id: 8fbaa844-5c7f-4d9f-b667-ac0ce0c88d90 2023-10-11T11:00:45.719 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring primary: true 2023-10-11T11:00:45.720 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n '' ']' 2023-10-11T11:00:45.720 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ stress_write_image cluster2 mirror image_42 2023-10-11T11:00:45.720 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:00:45.720 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:00:45.720 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_42 2023-10-11T11:00:45.721 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-10-11T11:00:45.721 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local duration=7 2023-10-11T11:00:45.721 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ set +e 2023-10-11T11:00:45.722 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_42 --debug-rbd=20 --debug-journaler=20 2023-10-11T11:00:46.002 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:100 IOs, 51910656 bytes 2023-10-11T11:00:46.120 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:200 IOs, 54860800 bytes 2023-10-11T11:00:46.262 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:300 IOs, 76261376 bytes 2023-10-11T11:00:46.423 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:400 IOs, 105536512 bytes 2023-10-11T11:00:46.553 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:500 IOs, 118061056 bytes 2023-10-11T11:00:46.790 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:600 IOs, 206016512 bytes 2023-10-11T11:00:46.932 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:700 IOs, 222759936 bytes 2023-10-11T11:00:47.193 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:800 IOs, 297907200 bytes 2023-10-11T11:00:47.432 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:900 IOs, 354796544 bytes 2023-10-11T11:00:47.583 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1000 IOs, 396219392 bytes 2023-10-11T11:00:47.787 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1100 IOs, 432310272 bytes 2023-10-11T11:00:47.921 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1200 IOs, 454784000 bytes 2023-10-11T11:00:48.052 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1300 IOs, 477419520 bytes 2023-10-11T11:00:48.247 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1400 IOs, 525594624 bytes 2023-10-11T11:00:48.414 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1500 IOs, 553676800 bytes 2023-10-11T11:00:48.571 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1600 IOs, 579048448 bytes 2023-10-11T11:00:48.782 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1700 IOs, 632653824 bytes 2023-10-11T11:00:49.039 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1800 IOs, 711865344 bytes 2023-10-11T11:00:49.278 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1900 IOs, 795222016 bytes 2023-10-11T11:00:49.482 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2000 IOs, 832474112 bytes 2023-10-11T11:00:49.655 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2100 IOs, 870341632 bytes 2023-10-11T11:00:49.806 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2200 IOs, 881653760 bytes 2023-10-11T11:00:49.945 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2300 IOs, 893735936 bytes 2023-10-11T11:00:50.075 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2400 IOs, 908875776 bytes 2023-10-11T11:00:50.226 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2500 IOs, 933803008 bytes 2023-10-11T11:00:50.417 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2600 IOs, 974155776 bytes 2023-10-11T11:00:50.675 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2700 IOs, 1048238080 bytes 2023-10-11T11:00:50.944 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2800 IOs, 1127372800 bytes 2023-10-11T11:00:51.145 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2900 IOs, 1176083456 bytes 2023-10-11T11:00:51.346 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3000 IOs, 1224296448 bytes 2023-10-11T11:00:51.585 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3100 IOs, 1287120896 bytes 2023-10-11T11:00:51.757 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3200 IOs, 1323381760 bytes 2023-10-11T11:00:51.938 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3300 IOs, 1358659584 bytes 2023-10-11T11:00:52.099 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3400 IOs, 1387947008 bytes 2023-10-11T11:00:52.329 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3500 IOs, 1452002304 bytes 2023-10-11T11:00:52.446 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3600 IOs, 1459090432 bytes 2023-10-11T11:00:52.567 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3700 IOs, 1478439936 bytes 2023-10-11T11:00:52.727 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ error_code=124 2023-10-11T11:00:52.727 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ set -e 2023-10-11T11:00:52.727 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' 124 -eq 124 ']' 2023-10-11T11:00:52.727 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:00:52.728 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T11:00:52.728 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_43 2023-10-11T11:00:52.728 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ create_image_and_enable_mirror cluster2 mirror image_43 snapshot 128M 2023-10-11T11:00:52.728 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:00:52.728 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T11:00:52.729 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:00:52.729 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T11:00:52.729 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_43 2023-10-11T11:00:52.729 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T11:00:52.729 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local mode=snapshot 2023-10-11T11:00:52.729 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n snapshot ']' 2023-10-11T11:00:52.730 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T11:00:52.730 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ create_image cluster2 mirror image_43 128M 2023-10-11T11:00:52.730 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:00:52.730 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T11:00:52.730 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:00:52.730 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T11:00:52.731 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_43 2023-10-11T11:00:52.731 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T11:00:52.731 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local size=128 2023-10-11T11:00:52.731 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n 128M ']' 2023-10-11T11:00:52.731 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ size=128M 2023-10-11T11:00:52.732 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T11:00:52.732 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_43 2023-10-11T11:00:52.776 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' image = image ']' 2023-10-11T11:00:52.777 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ enable_mirror cluster2 mirror image_43 snapshot 2023-10-11T11:00:52.777 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:00:52.777 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:00:52.777 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_43 2023-10-11T11:00:52.777 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local mode=snapshot 2023-10-11T11:00:52.777 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_43 snapshot 2023-10-11T11:00:53.718 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:Mirroring enabled 2023-10-11T11:00:53.730 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 info mirror/image_43 2023-10-11T11:00:53.768 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:rbd image 'image_43': 2023-10-11T11:00:53.769 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: size 128 MiB in 32 objects 2023-10-11T11:00:53.769 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: order 22 (4 MiB objects) 2023-10-11T11:00:53.769 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: snapshot_count: 1 2023-10-11T11:00:53.769 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: id: 13e840193e96 2023-10-11T11:00:53.769 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: block_name_prefix: rbd_data.13e840193e96 2023-10-11T11:00:53.770 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: format: 2 2023-10-11T11:00:53.770 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-10-11T11:00:53.770 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: op_features: 2023-10-11T11:00:53.770 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: flags: 2023-10-11T11:00:53.770 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: create_timestamp: Wed Oct 11 11:00:52 2023 2023-10-11T11:00:53.771 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: access_timestamp: Wed Oct 11 11:00:52 2023 2023-10-11T11:00:53.771 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: modify_timestamp: Wed Oct 11 11:00:52 2023 2023-10-11T11:00:53.771 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring state: enabled 2023-10-11T11:00:53.771 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring mode: snapshot 2023-10-11T11:00:53.771 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring global id: 008ad2e7-4a4a-41fc-aeec-300676cc4f95 2023-10-11T11:00:53.771 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring primary: true 2023-10-11T11:00:53.772 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n '' ']' 2023-10-11T11:00:53.773 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ stress_write_image cluster2 mirror image_43 2023-10-11T11:00:53.773 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:00:53.773 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:00:53.773 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_43 2023-10-11T11:00:53.773 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-10-11T11:00:53.774 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local duration=14 2023-10-11T11:00:53.774 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ set +e 2023-10-11T11:00:53.774 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_43 --debug-rbd=20 --debug-journaler=20 2023-10-11T11:00:54.030 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:100 IOs, 34432000 bytes 2023-10-11T11:00:54.263 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:200 IOs, 95592448 bytes 2023-10-11T11:00:54.478 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:300 IOs, 159980544 bytes 2023-10-11T11:00:54.692 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:400 IOs, 217291776 bytes 2023-10-11T11:00:54.866 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:500 IOs, 253032448 bytes 2023-10-11T11:00:55.117 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:600 IOs, 327197696 bytes 2023-10-11T11:00:55.293 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:700 IOs, 374241280 bytes 2023-10-11T11:00:55.530 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:800 IOs, 426489856 bytes 2023-10-11T11:00:55.771 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:900 IOs, 489922560 bytes 2023-10-11T11:00:55.952 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1000 IOs, 539823104 bytes 2023-10-11T11:00:56.177 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1100 IOs, 589114368 bytes 2023-10-11T11:00:56.345 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1200 IOs, 620496896 bytes 2023-10-11T11:00:56.654 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1300 IOs, 687844352 bytes 2023-10-11T11:00:56.775 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1400 IOs, 700442624 bytes 2023-10-11T11:00:56.998 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1500 IOs, 766251008 bytes 2023-10-11T11:00:57.207 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1600 IOs, 816636928 bytes 2023-10-11T11:00:57.406 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1700 IOs, 867798016 bytes 2023-10-11T11:00:57.593 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1800 IOs, 903312384 bytes 2023-10-11T11:00:57.745 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1900 IOs, 947086336 bytes 2023-10-11T11:00:57.943 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2000 IOs, 974880768 bytes 2023-10-11T11:00:58.210 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2100 IOs, 1035553792 bytes 2023-10-11T11:00:58.437 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2200 IOs, 1103731712 bytes 2023-10-11T11:00:58.783 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2300 IOs, 1220608000 bytes 2023-10-11T11:00:58.947 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2400 IOs, 1267501056 bytes 2023-10-11T11:00:59.204 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2500 IOs, 1344629760 bytes 2023-10-11T11:00:59.458 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2600 IOs, 1403312128 bytes 2023-10-11T11:00:59.747 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2700 IOs, 1509801984 bytes 2023-10-11T11:00:59.948 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2800 IOs, 1554295808 bytes 2023-10-11T11:01:00.138 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2900 IOs, 1600608256 bytes 2023-10-11T11:01:00.288 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3000 IOs, 1630316544 bytes 2023-10-11T11:01:00.574 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3100 IOs, 1729829888 bytes 2023-10-11T11:01:00.724 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3200 IOs, 1740858368 bytes 2023-10-11T11:01:00.917 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3300 IOs, 1785980928 bytes 2023-10-11T11:01:01.038 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3400 IOs, 1819193344 bytes 2023-10-11T11:01:01.266 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3500 IOs, 1857757184 bytes 2023-10-11T11:01:01.390 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3600 IOs, 1868696576 bytes 2023-10-11T11:01:01.548 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3700 IOs, 1895205888 bytes 2023-10-11T11:01:01.710 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3800 IOs, 1922909184 bytes 2023-10-11T11:01:01.871 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3900 IOs, 1951385600 bytes 2023-10-11T11:01:02.021 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4000 IOs, 1974345728 bytes 2023-10-11T11:01:02.201 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4100 IOs, 2018256896 bytes 2023-10-11T11:01:02.405 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4200 IOs, 2071417856 bytes 2023-10-11T11:01:02.558 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4300 IOs, 2096929792 bytes 2023-10-11T11:01:02.735 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4400 IOs, 2143273984 bytes 2023-10-11T11:01:03.091 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4500 IOs, 2251677696 bytes 2023-10-11T11:01:03.330 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4600 IOs, 2320067584 bytes 2023-10-11T11:01:03.581 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4700 IOs, 2395032576 bytes 2023-10-11T11:01:03.901 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4800 IOs, 2492706816 bytes 2023-10-11T11:01:04.088 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4900 IOs, 2540820480 bytes 2023-10-11T11:01:04.187 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5000 IOs, 2547306496 bytes 2023-10-11T11:01:04.327 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5100 IOs, 2571592704 bytes 2023-10-11T11:01:04.464 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5200 IOs, 2593745920 bytes 2023-10-11T11:01:04.733 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5300 IOs, 2679420928 bytes 2023-10-11T11:01:04.883 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5400 IOs, 2698642432 bytes 2023-10-11T11:01:05.082 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5500 IOs, 2749077504 bytes 2023-10-11T11:01:05.286 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5600 IOs, 2801378304 bytes 2023-10-11T11:01:05.460 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5700 IOs, 2832081920 bytes 2023-10-11T11:01:05.694 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5800 IOs, 2903636992 bytes 2023-10-11T11:01:05.867 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5900 IOs, 2931500032 bytes 2023-10-11T11:01:06.196 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6000 IOs, 3013617664 bytes 2023-10-11T11:01:06.408 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6100 IOs, 3076535296 bytes 2023-10-11T11:01:06.578 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6200 IOs, 3106609152 bytes 2023-10-11T11:01:06.732 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6300 IOs, 3128105984 bytes 2023-10-11T11:01:06.916 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6400 IOs, 3168401408 bytes 2023-10-11T11:01:07.161 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6500 IOs, 3230903296 bytes 2023-10-11T11:01:07.334 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6600 IOs, 3264072704 bytes 2023-10-11T11:01:07.511 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6700 IOs, 3298955264 bytes 2023-10-11T11:01:07.717 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6800 IOs, 3349217280 bytes 2023-10-11T11:01:07.780 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ error_code=124 2023-10-11T11:01:07.780 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ set -e 2023-10-11T11:01:07.780 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' 124 -eq 124 ']' 2023-10-11T11:01:07.781 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:01:07.781 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T11:01:07.781 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_44 2023-10-11T11:01:07.781 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ create_image_and_enable_mirror cluster2 mirror image_44 snapshot 128M 2023-10-11T11:01:07.781 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:01:07.782 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T11:01:07.782 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:01:07.782 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T11:01:07.782 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_44 2023-10-11T11:01:07.782 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T11:01:07.783 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local mode=snapshot 2023-10-11T11:01:07.783 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n snapshot ']' 2023-10-11T11:01:07.783 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T11:01:07.783 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ create_image cluster2 mirror image_44 128M 2023-10-11T11:01:07.783 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:01:07.784 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T11:01:07.784 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:01:07.784 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T11:01:07.784 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_44 2023-10-11T11:01:07.784 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T11:01:07.784 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local size=128 2023-10-11T11:01:07.785 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n 128M ']' 2023-10-11T11:01:07.785 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ size=128M 2023-10-11T11:01:07.785 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T11:01:07.785 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_44 2023-10-11T11:01:07.831 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' image = image ']' 2023-10-11T11:01:07.831 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ enable_mirror cluster2 mirror image_44 snapshot 2023-10-11T11:01:07.831 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:01:07.831 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:01:07.832 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_44 2023-10-11T11:01:07.832 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local mode=snapshot 2023-10-11T11:01:07.832 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_44 snapshot 2023-10-11T11:01:08.776 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:Mirroring enabled 2023-10-11T11:01:08.786 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 info mirror/image_44 2023-10-11T11:01:08.822 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:rbd image 'image_44': 2023-10-11T11:01:08.822 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: size 128 MiB in 32 objects 2023-10-11T11:01:08.822 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: order 22 (4 MiB objects) 2023-10-11T11:01:08.823 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: snapshot_count: 1 2023-10-11T11:01:08.823 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: id: 13f49158cc18 2023-10-11T11:01:08.823 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: block_name_prefix: rbd_data.13f49158cc18 2023-10-11T11:01:08.823 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: format: 2 2023-10-11T11:01:08.823 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-10-11T11:01:08.823 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: op_features: 2023-10-11T11:01:08.824 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: flags: 2023-10-11T11:01:08.824 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: create_timestamp: Wed Oct 11 11:01:07 2023 2023-10-11T11:01:08.824 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: access_timestamp: Wed Oct 11 11:01:07 2023 2023-10-11T11:01:08.824 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: modify_timestamp: Wed Oct 11 11:01:07 2023 2023-10-11T11:01:08.824 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring state: enabled 2023-10-11T11:01:08.825 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring mode: snapshot 2023-10-11T11:01:08.825 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring global id: 749fd0b3-e9a4-4bac-bd26-1b56437c5b8f 2023-10-11T11:01:08.825 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring primary: true 2023-10-11T11:01:08.825 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n '' ']' 2023-10-11T11:01:08.826 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ stress_write_image cluster2 mirror image_44 2023-10-11T11:01:08.826 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:01:08.826 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:01:08.826 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_44 2023-10-11T11:01:08.826 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-10-11T11:01:08.827 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local duration=13 2023-10-11T11:01:08.827 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ set +e 2023-10-11T11:01:08.827 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_44 --debug-rbd=20 --debug-journaler=20 2023-10-11T11:01:09.016 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:100 IOs, 618496 bytes 2023-10-11T11:01:09.242 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:200 IOs, 82910208 bytes 2023-10-11T11:01:09.483 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:300 IOs, 124165120 bytes 2023-10-11T11:01:09.651 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:400 IOs, 158178304 bytes 2023-10-11T11:01:09.944 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:500 IOs, 246446080 bytes 2023-10-11T11:01:10.083 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:600 IOs, 272144384 bytes 2023-10-11T11:01:10.222 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:700 IOs, 277268480 bytes 2023-10-11T11:01:10.416 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:800 IOs, 316511232 bytes 2023-10-11T11:01:10.591 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:900 IOs, 345960448 bytes 2023-10-11T11:01:10.861 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1000 IOs, 399570944 bytes 2023-10-11T11:01:11.025 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1100 IOs, 425747456 bytes 2023-10-11T11:01:11.160 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1200 IOs, 444866560 bytes 2023-10-11T11:01:11.411 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1300 IOs, 514870272 bytes 2023-10-11T11:01:11.626 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1400 IOs, 573229056 bytes 2023-10-11T11:01:11.864 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1500 IOs, 650977280 bytes 2023-10-11T11:01:12.187 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1600 IOs, 703542272 bytes 2023-10-11T11:01:12.382 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1700 IOs, 749399040 bytes 2023-10-11T11:01:12.497 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1800 IOs, 753406976 bytes 2023-10-11T11:01:12.698 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1900 IOs, 804186112 bytes 2023-10-11T11:01:12.895 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2000 IOs, 869258240 bytes 2023-10-11T11:01:13.094 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2100 IOs, 889175040 bytes 2023-10-11T11:01:13.290 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2200 IOs, 934853632 bytes 2023-10-11T11:01:13.534 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2300 IOs, 1012931584 bytes 2023-10-11T11:01:13.680 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2400 IOs, 1036051456 bytes 2023-10-11T11:01:13.870 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2500 IOs, 1077105664 bytes 2023-10-11T11:01:14.083 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2600 IOs, 1127203840 bytes 2023-10-11T11:01:14.291 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2700 IOs, 1180754944 bytes 2023-10-11T11:01:14.427 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2800 IOs, 1195508736 bytes 2023-10-11T11:01:14.589 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2900 IOs, 1220330496 bytes 2023-10-11T11:01:14.755 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3000 IOs, 1246498816 bytes 2023-10-11T11:01:14.920 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3100 IOs, 1278039040 bytes 2023-10-11T11:01:15.120 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3200 IOs, 1347727360 bytes 2023-10-11T11:01:15.331 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3300 IOs, 1405598720 bytes 2023-10-11T11:01:15.516 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3400 IOs, 1437265920 bytes 2023-10-11T11:01:15.718 INFO:tasks.ceph.cluster2.osd.0.smithi111.stderr:2023-10-11T11:01:15.716+0000 7f80be969700 -1 reset not still connected to 0x5555d6e4f110 2023-10-11T11:01:15.789 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3500 IOs, 1506740224 bytes 2023-10-11T11:01:16.017 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3600 IOs, 1554169856 bytes 2023-10-11T11:01:16.183 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3700 IOs, 1603383296 bytes 2023-10-11T11:01:16.418 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3800 IOs, 1644952576 bytes 2023-10-11T11:01:16.586 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3900 IOs, 1676053504 bytes 2023-10-11T11:01:16.849 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4000 IOs, 1762923520 bytes 2023-10-11T11:01:17.095 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4100 IOs, 1832011776 bytes 2023-10-11T11:01:17.303 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4200 IOs, 1880513536 bytes 2023-10-11T11:01:17.461 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4300 IOs, 1906855936 bytes 2023-10-11T11:01:17.609 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4400 IOs, 1926796288 bytes 2023-10-11T11:01:17.805 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4500 IOs, 1971877888 bytes 2023-10-11T11:01:17.990 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4600 IOs, 2015181824 bytes 2023-10-11T11:01:18.149 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4700 IOs, 2047776768 bytes 2023-10-11T11:01:18.268 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4800 IOs, 2055213056 bytes 2023-10-11T11:01:18.449 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4900 IOs, 2091284480 bytes 2023-10-11T11:01:18.719 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5000 IOs, 2174436352 bytes 2023-10-11T11:01:18.899 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5100 IOs, 2209084416 bytes 2023-10-11T11:01:19.141 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5200 IOs, 2277324800 bytes 2023-10-11T11:01:19.300 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5300 IOs, 2305557504 bytes 2023-10-11T11:01:19.450 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5400 IOs, 2325200896 bytes 2023-10-11T11:01:19.692 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5500 IOs, 2411433984 bytes 2023-10-11T11:01:19.898 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5600 IOs, 2433947648 bytes 2023-10-11T11:01:20.033 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5700 IOs, 2446256128 bytes 2023-10-11T11:01:20.161 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5800 IOs, 2459280384 bytes 2023-10-11T11:01:20.322 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5900 IOs, 2484176896 bytes 2023-10-11T11:01:20.504 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6000 IOs, 2534237184 bytes 2023-10-11T11:01:20.729 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6100 IOs, 2582676480 bytes 2023-10-11T11:01:20.875 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6200 IOs, 2609063936 bytes 2023-10-11T11:01:21.116 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6300 IOs, 2669088768 bytes 2023-10-11T11:01:21.286 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6400 IOs, 2692563968 bytes 2023-10-11T11:01:21.525 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6500 IOs, 2761686016 bytes 2023-10-11T11:01:21.668 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6600 IOs, 2780710912 bytes 2023-10-11T11:01:21.824 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6700 IOs, 2807604224 bytes 2023-10-11T11:01:21.834 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ error_code=124 2023-10-11T11:01:21.835 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ set -e 2023-10-11T11:01:21.835 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' 124 -eq 124 ']' 2023-10-11T11:01:21.835 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:01:21.835 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T11:01:21.836 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_45 2023-10-11T11:01:21.836 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ create_image_and_enable_mirror cluster2 mirror image_45 snapshot 128M 2023-10-11T11:01:21.836 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:01:21.836 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T11:01:21.836 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:01:21.837 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T11:01:21.837 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_45 2023-10-11T11:01:21.837 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T11:01:21.837 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local mode=snapshot 2023-10-11T11:01:21.837 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n snapshot ']' 2023-10-11T11:01:21.838 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T11:01:21.838 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ create_image cluster2 mirror image_45 128M 2023-10-11T11:01:21.838 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:01:21.838 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T11:01:21.838 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:01:21.839 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T11:01:21.839 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_45 2023-10-11T11:01:21.839 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T11:01:21.839 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local size=128 2023-10-11T11:01:21.839 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n 128M ']' 2023-10-11T11:01:21.839 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ size=128M 2023-10-11T11:01:21.840 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T11:01:21.840 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_45 2023-10-11T11:01:21.883 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' image = image ']' 2023-10-11T11:01:21.883 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ enable_mirror cluster2 mirror image_45 snapshot 2023-10-11T11:01:21.883 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:01:21.883 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:01:21.884 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_45 2023-10-11T11:01:21.884 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local mode=snapshot 2023-10-11T11:01:21.884 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_45 snapshot 2023-10-11T11:01:22.482 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:Mirroring enabled 2023-10-11T11:01:22.493 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 info mirror/image_45 2023-10-11T11:01:22.529 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:rbd image 'image_45': 2023-10-11T11:01:22.529 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: size 128 MiB in 32 objects 2023-10-11T11:01:22.529 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: order 22 (4 MiB objects) 2023-10-11T11:01:22.530 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: snapshot_count: 1 2023-10-11T11:01:22.530 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: id: 1400b0ce0eb1 2023-10-11T11:01:22.530 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: block_name_prefix: rbd_data.1400b0ce0eb1 2023-10-11T11:01:22.530 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: format: 2 2023-10-11T11:01:22.530 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-10-11T11:01:22.531 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: op_features: 2023-10-11T11:01:22.531 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: flags: 2023-10-11T11:01:22.531 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: create_timestamp: Wed Oct 11 11:01:21 2023 2023-10-11T11:01:22.531 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: access_timestamp: Wed Oct 11 11:01:21 2023 2023-10-11T11:01:22.531 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: modify_timestamp: Wed Oct 11 11:01:21 2023 2023-10-11T11:01:22.531 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring state: enabled 2023-10-11T11:01:22.532 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring mode: snapshot 2023-10-11T11:01:22.532 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring global id: fb355421-cd73-48b6-8cc6-59fefdadb677 2023-10-11T11:01:22.532 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring primary: true 2023-10-11T11:01:22.533 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n '' ']' 2023-10-11T11:01:22.533 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ stress_write_image cluster2 mirror image_45 2023-10-11T11:01:22.533 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:01:22.533 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:01:22.533 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_45 2023-10-11T11:01:22.534 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-10-11T11:01:22.534 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local duration=6 2023-10-11T11:01:22.534 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ set +e 2023-10-11T11:01:22.535 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_45 --debug-rbd=20 --debug-journaler=20 2023-10-11T11:01:22.780 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:100 IOs, 25902080 bytes 2023-10-11T11:01:22.928 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:200 IOs, 45890560 bytes 2023-10-11T11:01:23.049 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:300 IOs, 46525440 bytes 2023-10-11T11:01:23.266 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:400 IOs, 103514112 bytes 2023-10-11T11:01:23.485 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:500 IOs, 180723712 bytes 2023-10-11T11:01:23.765 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:600 IOs, 251395072 bytes 2023-10-11T11:01:23.885 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:700 IOs, 257201152 bytes 2023-10-11T11:01:24.066 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:800 IOs, 291261440 bytes 2023-10-11T11:01:24.218 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:900 IOs, 308116480 bytes 2023-10-11T11:01:24.389 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1000 IOs, 356193280 bytes 2023-10-11T11:01:24.569 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1100 IOs, 389533696 bytes 2023-10-11T11:01:24.750 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1200 IOs, 411558912 bytes 2023-10-11T11:01:24.904 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1300 IOs, 438638592 bytes 2023-10-11T11:01:25.044 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1400 IOs, 456871936 bytes 2023-10-11T11:01:25.242 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1500 IOs, 511900672 bytes 2023-10-11T11:01:25.370 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1600 IOs, 527092736 bytes 2023-10-11T11:01:25.514 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1700 IOs, 549235712 bytes 2023-10-11T11:01:25.668 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1800 IOs, 573447168 bytes 2023-10-11T11:01:25.822 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1900 IOs, 588194816 bytes 2023-10-11T11:01:25.990 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2000 IOs, 618387456 bytes 2023-10-11T11:01:26.148 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2100 IOs, 645853184 bytes 2023-10-11T11:01:26.377 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2200 IOs, 709876736 bytes 2023-10-11T11:01:26.615 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2300 IOs, 779873280 bytes 2023-10-11T11:01:26.820 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2400 IOs, 810703872 bytes 2023-10-11T11:01:26.954 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2500 IOs, 836813824 bytes 2023-10-11T11:01:27.186 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2600 IOs, 884247552 bytes 2023-10-11T11:01:27.365 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2700 IOs, 924512256 bytes 2023-10-11T11:01:27.470 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2800 IOs, 925106176 bytes 2023-10-11T11:01:27.623 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2900 IOs, 951116800 bytes 2023-10-11T11:01:27.799 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3000 IOs, 986085376 bytes 2023-10-11T11:01:27.914 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3100 IOs, 986691584 bytes 2023-10-11T11:01:28.080 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3200 IOs, 1009935360 bytes 2023-10-11T11:01:28.240 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3300 IOs, 1035514880 bytes 2023-10-11T11:01:28.480 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3400 IOs, 1102399488 bytes 2023-10-11T11:01:28.540 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ error_code=124 2023-10-11T11:01:28.540 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ set -e 2023-10-11T11:01:28.540 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' 124 -eq 124 ']' 2023-10-11T11:01:28.541 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:01:28.541 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T11:01:28.541 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_46 2023-10-11T11:01:28.541 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ create_image_and_enable_mirror cluster2 mirror image_46 snapshot 128M 2023-10-11T11:01:28.541 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:01:28.541 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T11:01:28.542 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:01:28.542 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T11:01:28.542 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_46 2023-10-11T11:01:28.542 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T11:01:28.542 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local mode=snapshot 2023-10-11T11:01:28.543 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n snapshot ']' 2023-10-11T11:01:28.543 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T11:01:28.543 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ create_image cluster2 mirror image_46 128M 2023-10-11T11:01:28.543 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:01:28.543 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T11:01:28.544 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:01:28.544 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T11:01:28.544 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_46 2023-10-11T11:01:28.544 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T11:01:28.544 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local size=128 2023-10-11T11:01:28.545 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n 128M ']' 2023-10-11T11:01:28.545 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ size=128M 2023-10-11T11:01:28.545 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T11:01:28.545 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_46 2023-10-11T11:01:28.588 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' image = image ']' 2023-10-11T11:01:28.588 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ enable_mirror cluster2 mirror image_46 snapshot 2023-10-11T11:01:28.588 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:01:28.588 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:01:28.589 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_46 2023-10-11T11:01:28.589 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local mode=snapshot 2023-10-11T11:01:28.589 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_46 snapshot 2023-10-11T11:01:29.508 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:Mirroring enabled 2023-10-11T11:01:29.518 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 info mirror/image_46 2023-10-11T11:01:29.556 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:rbd image 'image_46': 2023-10-11T11:01:29.556 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: size 128 MiB in 32 objects 2023-10-11T11:01:29.556 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: order 22 (4 MiB objects) 2023-10-11T11:01:29.556 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: snapshot_count: 1 2023-10-11T11:01:29.557 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: id: 140c143a092 2023-10-11T11:01:29.557 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: block_name_prefix: rbd_data.140c143a092 2023-10-11T11:01:29.557 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: format: 2 2023-10-11T11:01:29.557 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-10-11T11:01:29.557 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: op_features: 2023-10-11T11:01:29.557 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: flags: 2023-10-11T11:01:29.558 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: create_timestamp: Wed Oct 11 11:01:28 2023 2023-10-11T11:01:29.558 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: access_timestamp: Wed Oct 11 11:01:28 2023 2023-10-11T11:01:29.558 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: modify_timestamp: Wed Oct 11 11:01:28 2023 2023-10-11T11:01:29.558 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring state: enabled 2023-10-11T11:01:29.558 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring mode: snapshot 2023-10-11T11:01:29.559 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring global id: 96075109-54b3-44cf-8c55-4ffc63857fb0 2023-10-11T11:01:29.559 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring primary: true 2023-10-11T11:01:29.559 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n '' ']' 2023-10-11T11:01:29.560 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ stress_write_image cluster2 mirror image_46 2023-10-11T11:01:29.560 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:01:29.560 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:01:29.560 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_46 2023-10-11T11:01:29.560 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-10-11T11:01:29.561 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local duration=10 2023-10-11T11:01:29.561 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ set +e 2023-10-11T11:01:29.561 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_46 --debug-rbd=20 --debug-journaler=20 2023-10-11T11:01:29.751 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:100 IOs, 2177024 bytes 2023-10-11T11:01:30.004 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:200 IOs, 72914944 bytes 2023-10-11T11:01:30.243 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:300 IOs, 146626560 bytes 2023-10-11T11:01:30.452 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:400 IOs, 225064960 bytes 2023-10-11T11:01:30.680 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:500 IOs, 258055168 bytes 2023-10-11T11:01:30.894 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:600 IOs, 306735104 bytes 2023-10-11T11:01:31.051 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:700 IOs, 338817024 bytes 2023-10-11T11:01:31.181 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:800 IOs, 365840384 bytes 2023-10-11T11:01:31.338 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:900 IOs, 386438144 bytes 2023-10-11T11:01:31.549 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1000 IOs, 432559104 bytes 2023-10-11T11:01:31.789 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1100 IOs, 514436096 bytes 2023-10-11T11:01:31.970 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1200 IOs, 539720704 bytes 2023-10-11T11:01:32.136 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1300 IOs, 591452160 bytes 2023-10-11T11:01:32.418 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1400 IOs, 661387264 bytes 2023-10-11T11:01:32.617 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1500 IOs, 714607616 bytes 2023-10-11T11:01:32.793 INFO:tasks.ceph.cluster2.osd.0.smithi111.stderr:2023-10-11T11:01:32.792+0000 7f80be969700 -1 reset not still connected to 0x5555fe9c2820 2023-10-11T11:01:32.793 INFO:tasks.ceph.cluster2.osd.0.smithi111.stderr:2023-10-11T11:01:32.792+0000 7f80be969700 -1 reset not still connected to 0x55560047e1a0 2023-10-11T11:01:32.793 INFO:tasks.ceph.cluster2.osd.0.smithi111.stderr:2023-10-11T11:01:32.792+0000 7f80be969700 -1 reset not still connected to 0x555600dee270 2023-10-11T11:01:32.906 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1600 IOs, 794014720 bytes 2023-10-11T11:01:33.128 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1700 IOs, 853931008 bytes 2023-10-11T11:01:33.330 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1800 IOs, 906942464 bytes 2023-10-11T11:01:33.441 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1900 IOs, 907551744 bytes 2023-10-11T11:01:33.550 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2000 IOs, 911364096 bytes 2023-10-11T11:01:33.555 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.8224168009629614) > weight (0.25) 2023-10-11T11:01:33.555 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.3372481094023496) > weight (0.25) 2023-10-11T11:01:33.556 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.3689372494327485) > weight (0.25) 2023-10-11T11:01:33.556 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.27317546461276765) > weight (0.25) 2023-10-11T11:01:33.556 INFO:tasks.rbd_mirror_thrash:waiting for 111 secs before thrashing 2023-10-11T11:01:33.733 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2100 IOs, 954665984 bytes 2023-10-11T11:01:33.860 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2200 IOs, 978360320 bytes 2023-10-11T11:01:33.994 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2300 IOs, 978966528 bytes 2023-10-11T11:01:34.255 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2400 IOs, 1045967872 bytes 2023-10-11T11:01:34.466 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2500 IOs, 1094841344 bytes 2023-10-11T11:01:34.600 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2600 IOs, 1112207360 bytes 2023-10-11T11:01:34.746 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2700 IOs, 1136062464 bytes 2023-10-11T11:01:34.903 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2800 IOs, 1162162176 bytes 2023-10-11T11:01:35.079 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2900 IOs, 1207252992 bytes 2023-10-11T11:01:35.212 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3000 IOs, 1217278976 bytes 2023-10-11T11:01:35.422 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3100 IOs, 1270231040 bytes 2023-10-11T11:01:35.641 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3200 IOs, 1327738880 bytes 2023-10-11T11:01:35.832 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3300 IOs, 1372279808 bytes 2023-10-11T11:01:35.964 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3400 IOs, 1387790336 bytes 2023-10-11T11:01:36.219 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3500 IOs, 1464443904 bytes 2023-10-11T11:01:36.362 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3600 IOs, 1482359808 bytes 2023-10-11T11:01:36.714 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3700 IOs, 1606571008 bytes 2023-10-11T11:01:36.897 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3800 IOs, 1648295936 bytes 2023-10-11T11:01:37.053 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3900 IOs, 1674514432 bytes 2023-10-11T11:01:37.208 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4000 IOs, 1698037760 bytes 2023-10-11T11:01:37.413 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4100 IOs, 1751112704 bytes 2023-10-11T11:01:37.741 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4200 IOs, 1870928896 bytes 2023-10-11T11:01:37.914 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4300 IOs, 1904295936 bytes 2023-10-11T11:01:38.035 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4400 IOs, 1912678400 bytes 2023-10-11T11:01:38.159 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4500 IOs, 1918497792 bytes 2023-10-11T11:01:38.526 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4600 IOs, 2049205248 bytes 2023-10-11T11:01:38.638 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4700 IOs, 2056440832 bytes 2023-10-11T11:01:38.773 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4800 IOs, 2075750400 bytes 2023-10-11T11:01:38.943 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4900 IOs, 2104906752 bytes 2023-10-11T11:01:39.104 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5000 IOs, 2131912704 bytes 2023-10-11T11:01:39.255 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5100 IOs, 2152374272 bytes 2023-10-11T11:01:39.454 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5200 IOs, 2201163776 bytes 2023-10-11T11:01:39.570 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ error_code=124 2023-10-11T11:01:39.571 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ set -e 2023-10-11T11:01:39.571 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' 124 -eq 124 ']' 2023-10-11T11:01:39.571 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:01:39.571 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T11:01:39.572 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_47 2023-10-11T11:01:39.572 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ create_image_and_enable_mirror cluster2 mirror image_47 snapshot 128M 2023-10-11T11:01:39.572 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:01:39.572 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T11:01:39.572 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:01:39.573 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T11:01:39.573 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_47 2023-10-11T11:01:39.573 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T11:01:39.573 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local mode=snapshot 2023-10-11T11:01:39.573 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n snapshot ']' 2023-10-11T11:01:39.574 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T11:01:39.574 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ create_image cluster2 mirror image_47 128M 2023-10-11T11:01:39.574 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:01:39.574 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T11:01:39.575 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:01:39.575 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T11:01:39.575 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_47 2023-10-11T11:01:39.575 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T11:01:39.575 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local size=128 2023-10-11T11:01:39.576 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n 128M ']' 2023-10-11T11:01:39.576 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ size=128M 2023-10-11T11:01:39.576 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T11:01:39.576 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_47 2023-10-11T11:01:39.620 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' image = image ']' 2023-10-11T11:01:39.621 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ enable_mirror cluster2 mirror image_47 snapshot 2023-10-11T11:01:39.621 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:01:39.621 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:01:39.621 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_47 2023-10-11T11:01:39.621 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local mode=snapshot 2023-10-11T11:01:39.622 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_47 snapshot 2023-10-11T11:01:40.578 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:Mirroring enabled 2023-10-11T11:01:40.593 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 info mirror/image_47 2023-10-11T11:01:40.630 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:rbd image 'image_47': 2023-10-11T11:01:40.631 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: size 128 MiB in 32 objects 2023-10-11T11:01:40.631 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: order 22 (4 MiB objects) 2023-10-11T11:01:40.631 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: snapshot_count: 1 2023-10-11T11:01:40.632 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: id: 141870f3b30 2023-10-11T11:01:40.632 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: block_name_prefix: rbd_data.141870f3b30 2023-10-11T11:01:40.632 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: format: 2 2023-10-11T11:01:40.632 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-10-11T11:01:40.633 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: op_features: 2023-10-11T11:01:40.633 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: flags: 2023-10-11T11:01:40.633 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: create_timestamp: Wed Oct 11 11:01:39 2023 2023-10-11T11:01:40.634 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: access_timestamp: Wed Oct 11 11:01:39 2023 2023-10-11T11:01:40.634 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: modify_timestamp: Wed Oct 11 11:01:39 2023 2023-10-11T11:01:40.634 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring state: enabled 2023-10-11T11:01:40.634 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring mode: snapshot 2023-10-11T11:01:40.635 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring global id: 76f1a297-bf03-44bd-a55f-4ceb8d92fdbd 2023-10-11T11:01:40.635 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring primary: true 2023-10-11T11:01:40.636 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n '' ']' 2023-10-11T11:01:40.637 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ stress_write_image cluster2 mirror image_47 2023-10-11T11:01:40.637 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:01:40.637 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:01:40.638 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_47 2023-10-11T11:01:40.638 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-10-11T11:01:40.638 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local duration=8 2023-10-11T11:01:40.639 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ set +e 2023-10-11T11:01:40.639 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_47 --debug-rbd=20 --debug-journaler=20 2023-10-11T11:01:40.861 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:100 IOs, 23053312 bytes 2023-10-11T11:01:41.055 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:200 IOs, 67469312 bytes 2023-10-11T11:01:41.217 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:300 IOs, 90943488 bytes 2023-10-11T11:01:41.461 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:400 IOs, 154644480 bytes 2023-10-11T11:01:41.761 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:500 IOs, 259301376 bytes 2023-10-11T11:01:41.901 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:600 IOs, 279232512 bytes 2023-10-11T11:01:42.103 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:700 IOs, 323699712 bytes 2023-10-11T11:01:42.356 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:800 IOs, 351710208 bytes 2023-10-11T11:01:42.970 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:900 IOs, 384359424 bytes 2023-10-11T11:01:43.128 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1000 IOs, 393722880 bytes 2023-10-11T11:01:43.300 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1100 IOs, 428792832 bytes 2023-10-11T11:01:43.529 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1200 IOs, 457466880 bytes 2023-10-11T11:01:43.652 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1300 IOs, 465059840 bytes 2023-10-11T11:01:43.803 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1400 IOs, 488530944 bytes 2023-10-11T11:01:43.997 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1500 IOs, 533534720 bytes 2023-10-11T11:01:44.182 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1600 IOs, 573399040 bytes 2023-10-11T11:01:44.387 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1700 IOs, 618966016 bytes 2023-10-11T11:01:44.531 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1800 IOs, 645421056 bytes 2023-10-11T11:01:44.683 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1900 IOs, 662577152 bytes 2023-10-11T11:01:44.869 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2000 IOs, 705595392 bytes 2023-10-11T11:01:44.991 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2100 IOs, 714393600 bytes 2023-10-11T11:01:45.164 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2200 IOs, 750383104 bytes 2023-10-11T11:01:45.477 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2300 IOs, 839149568 bytes 2023-10-11T11:01:45.727 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2400 IOs, 907893760 bytes 2023-10-11T11:01:45.852 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2500 IOs, 923070464 bytes 2023-10-11T11:01:46.020 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2600 IOs, 967817216 bytes 2023-10-11T11:01:46.247 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2700 IOs, 1007930368 bytes 2023-10-11T11:01:46.464 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2800 IOs, 1059588096 bytes 2023-10-11T11:01:46.673 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2900 IOs, 1121599488 bytes 2023-10-11T11:01:46.823 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3000 IOs, 1136371712 bytes 2023-10-11T11:01:46.982 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3100 IOs, 1169177600 bytes 2023-10-11T11:01:47.140 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3200 IOs, 1199293440 bytes 2023-10-11T11:01:47.272 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3300 IOs, 1209516032 bytes 2023-10-11T11:01:47.466 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3400 IOs, 1259810816 bytes 2023-10-11T11:01:47.746 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3500 IOs, 1344877568 bytes 2023-10-11T11:01:47.994 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3600 IOs, 1419174912 bytes 2023-10-11T11:01:48.179 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3700 IOs, 1449715712 bytes 2023-10-11T11:01:48.338 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3800 IOs, 1466998784 bytes 2023-10-11T11:01:48.479 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3900 IOs, 1479162880 bytes 2023-10-11T11:01:48.640 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ error_code=124 2023-10-11T11:01:48.641 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ set -e 2023-10-11T11:01:48.641 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' 124 -eq 124 ']' 2023-10-11T11:01:48.641 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:01:48.641 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T11:01:48.641 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_48 2023-10-11T11:01:48.642 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ create_image_and_enable_mirror cluster2 mirror image_48 snapshot 128M 2023-10-11T11:01:48.642 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:01:48.642 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T11:01:48.642 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:01:48.642 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T11:01:48.643 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_48 2023-10-11T11:01:48.643 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T11:01:48.643 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local mode=snapshot 2023-10-11T11:01:48.643 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n snapshot ']' 2023-10-11T11:01:48.643 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T11:01:48.643 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ create_image cluster2 mirror image_48 128M 2023-10-11T11:01:48.644 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:01:48.644 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T11:01:48.644 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:01:48.644 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T11:01:48.644 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_48 2023-10-11T11:01:48.645 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T11:01:48.645 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local size=128 2023-10-11T11:01:48.645 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n 128M ']' 2023-10-11T11:01:48.645 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ size=128M 2023-10-11T11:01:48.645 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T11:01:48.645 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_48 2023-10-11T11:01:48.688 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' image = image ']' 2023-10-11T11:01:48.688 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ enable_mirror cluster2 mirror image_48 snapshot 2023-10-11T11:01:48.688 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:01:48.689 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:01:48.689 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_48 2023-10-11T11:01:48.689 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local mode=snapshot 2023-10-11T11:01:48.689 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_48 snapshot 2023-10-11T11:01:49.623 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:Mirroring enabled 2023-10-11T11:01:49.632 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 info mirror/image_48 2023-10-11T11:01:49.683 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:rbd image 'image_48': 2023-10-11T11:01:49.683 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: size 128 MiB in 32 objects 2023-10-11T11:01:49.683 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: order 22 (4 MiB objects) 2023-10-11T11:01:49.683 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: snapshot_count: 1 2023-10-11T11:01:49.684 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: id: 142414f2880d 2023-10-11T11:01:49.684 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: block_name_prefix: rbd_data.142414f2880d 2023-10-11T11:01:49.684 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: format: 2 2023-10-11T11:01:49.684 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-10-11T11:01:49.684 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: op_features: 2023-10-11T11:01:49.685 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: flags: 2023-10-11T11:01:49.685 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: create_timestamp: Wed Oct 11 11:01:48 2023 2023-10-11T11:01:49.685 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: access_timestamp: Wed Oct 11 11:01:48 2023 2023-10-11T11:01:49.685 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: modify_timestamp: Wed Oct 11 11:01:48 2023 2023-10-11T11:01:49.685 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring state: enabled 2023-10-11T11:01:49.686 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring mode: snapshot 2023-10-11T11:01:49.686 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring global id: a8b63bdf-0dc4-4ae7-bada-d821d5883be6 2023-10-11T11:01:49.686 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring primary: true 2023-10-11T11:01:49.687 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n '' ']' 2023-10-11T11:01:49.688 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ stress_write_image cluster2 mirror image_48 2023-10-11T11:01:49.688 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:01:49.688 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:01:49.688 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_48 2023-10-11T11:01:49.688 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-10-11T11:01:49.689 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local duration=10 2023-10-11T11:01:49.689 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ set +e 2023-10-11T11:01:49.689 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_48 --debug-rbd=20 --debug-journaler=20 2023-10-11T11:01:49.919 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:100 IOs, 32923648 bytes 2023-10-11T11:01:50.193 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:200 IOs, 96555008 bytes 2023-10-11T11:01:50.397 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:300 IOs, 154182656 bytes 2023-10-11T11:01:50.534 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:400 IOs, 179265536 bytes 2023-10-11T11:01:50.687 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:500 IOs, 201107456 bytes 2023-10-11T11:01:50.953 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:600 IOs, 274134016 bytes 2023-10-11T11:01:51.130 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:700 IOs, 307737600 bytes 2023-10-11T11:01:51.273 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:800 IOs, 351279104 bytes 2023-10-11T11:01:51.745 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:900 IOs, 437161984 bytes 2023-10-11T11:01:52.000 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1000 IOs, 514178048 bytes 2023-10-11T11:01:52.228 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1100 IOs, 583464960 bytes 2023-10-11T11:01:52.416 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1200 IOs, 623796224 bytes 2023-10-11T11:01:52.575 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1300 IOs, 662883328 bytes 2023-10-11T11:01:52.785 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1400 IOs, 706191360 bytes 2023-10-11T11:01:52.977 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1500 IOs, 759444480 bytes 2023-10-11T11:01:53.180 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1600 IOs, 783892480 bytes 2023-10-11T11:01:53.382 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1700 IOs, 833914880 bytes 2023-10-11T11:01:53.502 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1800 IOs, 840689664 bytes 2023-10-11T11:01:53.641 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1900 IOs, 860888064 bytes 2023-10-11T11:01:53.860 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2000 IOs, 915994624 bytes 2023-10-11T11:01:54.031 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2100 IOs, 970724352 bytes 2023-10-11T11:01:54.218 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2200 IOs, 992396288 bytes 2023-10-11T11:01:54.464 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2300 IOs, 1061000192 bytes 2023-10-11T11:01:54.654 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2400 IOs, 1102708736 bytes 2023-10-11T11:01:54.912 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2500 IOs, 1175508992 bytes 2023-10-11T11:01:55.181 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2600 IOs, 1255025664 bytes 2023-10-11T11:01:55.448 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2700 IOs, 1322425344 bytes 2023-10-11T11:01:55.577 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2800 IOs, 1333173248 bytes 2023-10-11T11:01:55.698 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2900 IOs, 1337223168 bytes 2023-10-11T11:01:55.887 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3000 IOs, 1373852672 bytes 2023-10-11T11:01:56.001 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3100 IOs, 1374484480 bytes 2023-10-11T11:01:56.168 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3200 IOs, 1420545024 bytes 2023-10-11T11:01:56.329 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3300 IOs, 1440096256 bytes 2023-10-11T11:01:56.560 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3400 IOs, 1508848640 bytes 2023-10-11T11:01:56.706 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3500 IOs, 1530567680 bytes 2023-10-11T11:01:56.869 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3600 IOs, 1563717632 bytes 2023-10-11T11:01:57.096 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3700 IOs, 1625483264 bytes 2023-10-11T11:01:57.389 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3800 IOs, 1721488384 bytes 2023-10-11T11:01:57.600 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3900 IOs, 1776454656 bytes 2023-10-11T11:01:57.802 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4000 IOs, 1823127552 bytes 2023-10-11T11:01:57.935 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4100 IOs, 1839848448 bytes 2023-10-11T11:01:58.085 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4200 IOs, 1861556224 bytes 2023-10-11T11:01:58.216 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4300 IOs, 1869385728 bytes 2023-10-11T11:01:58.393 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4400 IOs, 1907451904 bytes 2023-10-11T11:01:58.557 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4500 IOs, 1942227968 bytes 2023-10-11T11:01:58.756 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4600 IOs, 1990686720 bytes 2023-10-11T11:01:58.920 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4700 IOs, 2033700864 bytes 2023-10-11T11:01:59.090 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4800 IOs, 2056727552 bytes 2023-10-11T11:01:59.258 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4900 IOs, 2095840256 bytes 2023-10-11T11:01:59.409 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5000 IOs, 2129820672 bytes 2023-10-11T11:01:59.580 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5100 IOs, 2148182016 bytes 2023-10-11T11:01:59.697 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ error_code=124 2023-10-11T11:01:59.697 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ set -e 2023-10-11T11:01:59.698 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' 124 -eq 124 ']' 2023-10-11T11:01:59.698 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:01:59.698 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T11:01:59.698 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_49 2023-10-11T11:01:59.698 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ create_image_and_enable_mirror cluster2 mirror image_49 snapshot 128M 2023-10-11T11:01:59.698 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:01:59.699 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T11:01:59.699 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:01:59.699 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T11:01:59.699 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_49 2023-10-11T11:01:59.699 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T11:01:59.700 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local mode=snapshot 2023-10-11T11:01:59.700 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n snapshot ']' 2023-10-11T11:01:59.700 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T11:01:59.700 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ create_image cluster2 mirror image_49 128M 2023-10-11T11:01:59.700 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:01:59.701 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T11:01:59.701 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:01:59.701 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T11:01:59.701 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_49 2023-10-11T11:01:59.701 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T11:01:59.701 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local size=128 2023-10-11T11:01:59.702 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n 128M ']' 2023-10-11T11:01:59.702 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ size=128M 2023-10-11T11:01:59.702 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T11:01:59.702 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_49 2023-10-11T11:01:59.745 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' image = image ']' 2023-10-11T11:01:59.745 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ enable_mirror cluster2 mirror image_49 snapshot 2023-10-11T11:01:59.746 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:01:59.746 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:01:59.746 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_49 2023-10-11T11:01:59.746 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local mode=snapshot 2023-10-11T11:01:59.746 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_49 snapshot 2023-10-11T11:02:00.694 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:Mirroring enabled 2023-10-11T11:02:00.705 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 info mirror/image_49 2023-10-11T11:02:00.742 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:rbd image 'image_49': 2023-10-11T11:02:00.742 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: size 128 MiB in 32 objects 2023-10-11T11:02:00.743 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: order 22 (4 MiB objects) 2023-10-11T11:02:00.743 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: snapshot_count: 1 2023-10-11T11:02:00.743 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: id: 1430253deb5 2023-10-11T11:02:00.743 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: block_name_prefix: rbd_data.1430253deb5 2023-10-11T11:02:00.743 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: format: 2 2023-10-11T11:02:00.744 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-10-11T11:02:00.744 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: op_features: 2023-10-11T11:02:00.744 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: flags: 2023-10-11T11:02:00.744 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: create_timestamp: Wed Oct 11 11:01:59 2023 2023-10-11T11:02:00.744 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: access_timestamp: Wed Oct 11 11:01:59 2023 2023-10-11T11:02:00.744 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: modify_timestamp: Wed Oct 11 11:01:59 2023 2023-10-11T11:02:00.745 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring state: enabled 2023-10-11T11:02:00.745 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring mode: snapshot 2023-10-11T11:02:00.745 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring global id: 80e3527d-c1f3-41a7-917b-e93369287a62 2023-10-11T11:02:00.745 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring primary: true 2023-10-11T11:02:00.746 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n '' ']' 2023-10-11T11:02:00.746 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ stress_write_image cluster2 mirror image_49 2023-10-11T11:02:00.747 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:02:00.747 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:02:00.747 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_49 2023-10-11T11:02:00.747 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-10-11T11:02:00.748 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local duration=8 2023-10-11T11:02:00.748 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ set +e 2023-10-11T11:02:00.748 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_49 --debug-rbd=20 --debug-journaler=20 2023-10-11T11:02:01.006 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:100 IOs, 35591168 bytes 2023-10-11T11:02:01.277 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:200 IOs, 119239680 bytes 2023-10-11T11:02:01.402 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:300 IOs, 128999424 bytes 2023-10-11T11:02:01.508 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:400 IOs, 133469184 bytes 2023-10-11T11:02:01.649 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:500 IOs, 153814016 bytes 2023-10-11T11:02:01.834 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:600 IOs, 191783936 bytes 2023-10-11T11:02:01.943 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:700 IOs, 192419840 bytes 2023-10-11T11:02:02.112 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:800 IOs, 236760064 bytes 2023-10-11T11:02:02.358 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:900 IOs, 303373312 bytes 2023-10-11T11:02:02.612 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1000 IOs, 378814464 bytes 2023-10-11T11:02:02.827 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1100 IOs, 408316928 bytes 2023-10-11T11:02:03.006 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1200 IOs, 442514432 bytes 2023-10-11T11:02:03.179 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1300 IOs, 474918912 bytes 2023-10-11T11:02:03.315 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1400 IOs, 492831744 bytes 2023-10-11T11:02:03.459 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1500 IOs, 510098432 bytes 2023-10-11T11:02:03.705 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1600 IOs, 582044672 bytes 2023-10-11T11:02:03.836 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1700 IOs, 588699648 bytes 2023-10-11T11:02:04.026 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1800 IOs, 627037184 bytes 2023-10-11T11:02:04.184 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1900 IOs, 670662656 bytes 2023-10-11T11:02:04.532 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2000 IOs, 787805184 bytes 2023-10-11T11:02:04.726 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2100 IOs, 818166784 bytes 2023-10-11T11:02:04.863 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2200 IOs, 830980096 bytes 2023-10-11T11:02:05.037 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2300 IOs, 860015616 bytes 2023-10-11T11:02:05.172 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2400 IOs, 879523840 bytes 2023-10-11T11:02:05.397 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2500 IOs, 934409216 bytes 2023-10-11T11:02:05.676 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2600 IOs, 1014354944 bytes 2023-10-11T11:02:05.852 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2700 IOs, 1046057984 bytes 2023-10-11T11:02:05.994 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2800 IOs, 1059559424 bytes 2023-10-11T11:02:06.151 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2900 IOs, 1081445376 bytes 2023-10-11T11:02:06.307 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3000 IOs, 1107201024 bytes 2023-10-11T11:02:06.517 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3100 IOs, 1158445056 bytes 2023-10-11T11:02:06.694 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3200 IOs, 1192497152 bytes 2023-10-11T11:02:06.948 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3300 IOs, 1269069824 bytes 2023-10-11T11:02:07.080 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3400 IOs, 1278022656 bytes 2023-10-11T11:02:07.245 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3500 IOs, 1331269632 bytes 2023-10-11T11:02:07.529 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3600 IOs, 1396878336 bytes 2023-10-11T11:02:07.649 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3700 IOs, 1402078208 bytes 2023-10-11T11:02:07.804 INFO:tasks.ceph.cluster2.osd.0.smithi111.stderr:2023-10-11T11:02:07.797+0000 7f80be969700 -1 reset not still connected to 0x5555c93565b0 2023-10-11T11:02:07.804 INFO:tasks.ceph.cluster2.osd.0.smithi111.stderr:2023-10-11T11:02:07.797+0000 7f80be969700 -1 reset not still connected to 0x5555c9356d00 2023-10-11T11:02:07.805 INFO:tasks.ceph.cluster2.osd.0.smithi111.stderr:2023-10-11T11:02:07.797+0000 7f80be969700 -1 reset not still connected to 0x5555c9356dd0 2023-10-11T11:02:07.805 INFO:tasks.ceph.cluster2.osd.0.smithi111.stderr:2023-10-11T11:02:07.797+0000 7f80be969700 -1 reset not still connected to 0x5555c9356f70 2023-10-11T11:02:07.805 INFO:tasks.ceph.cluster2.osd.0.smithi111.stderr:2023-10-11T11:02:07.797+0000 7f80be969700 -1 reset not still connected to 0x5555c9357040 2023-10-11T11:02:07.805 INFO:tasks.ceph.cluster2.osd.0.smithi111.stderr:2023-10-11T11:02:07.797+0000 7f80be969700 -1 reset not still connected to 0x5555c9357110 2023-10-11T11:02:07.806 INFO:tasks.ceph.cluster2.osd.0.smithi111.stderr:2023-10-11T11:02:07.797+0000 7f80be969700 -1 reset not still connected to 0x5555c9357450 2023-10-11T11:02:07.806 INFO:tasks.ceph.cluster2.osd.0.smithi111.stderr:2023-10-11T11:02:07.797+0000 7f80be969700 -1 reset not still connected to 0x5555c93575f0 2023-10-11T11:02:07.806 INFO:tasks.ceph.cluster2.osd.0.smithi111.stderr:2023-10-11T11:02:07.797+0000 7f80be969700 -1 reset not still connected to 0x5555dd330820 2023-10-11T11:02:07.806 INFO:tasks.ceph.cluster2.osd.0.smithi111.stderr:2023-10-11T11:02:07.797+0000 7f80be969700 -1 reset not still connected to 0x5555e2c52340 2023-10-11T11:02:07.807 INFO:tasks.ceph.cluster2.osd.0.smithi111.stderr:2023-10-11T11:02:07.797+0000 7f80be969700 -1 reset not still connected to 0x5555e309f450 2023-10-11T11:02:07.807 INFO:tasks.ceph.cluster2.osd.0.smithi111.stderr:2023-10-11T11:02:07.797+0000 7f80be969700 -1 reset not still connected to 0x5555f5922340 2023-10-11T11:02:07.807 INFO:tasks.ceph.cluster2.osd.0.smithi111.stderr:2023-10-11T11:02:07.797+0000 7f80be969700 -1 reset not still connected to 0x5555fe9c2820 2023-10-11T11:02:07.807 INFO:tasks.ceph.cluster2.osd.0.smithi111.stderr:2023-10-11T11:02:07.799+0000 7f80be969700 -1 reset not still connected to 0x555600dee270 2023-10-11T11:02:07.883 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3800 IOs, 1470954496 bytes 2023-10-11T11:02:08.135 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3900 IOs, 1540710400 bytes 2023-10-11T11:02:08.266 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4000 IOs, 1550161920 bytes 2023-10-11T11:02:08.473 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4100 IOs, 1610655744 bytes 2023-10-11T11:02:08.759 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ error_code=124 2023-10-11T11:02:08.759 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ set -e 2023-10-11T11:02:08.760 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' 124 -eq 124 ']' 2023-10-11T11:02:08.760 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:02:08.760 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T11:02:08.761 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_50 2023-10-11T11:02:08.761 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ create_image_and_enable_mirror cluster2 mirror image_50 snapshot 128M 2023-10-11T11:02:08.761 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:02:08.761 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T11:02:08.762 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:02:08.762 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T11:02:08.762 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_50 2023-10-11T11:02:08.762 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T11:02:08.763 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local mode=snapshot 2023-10-11T11:02:08.763 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n snapshot ']' 2023-10-11T11:02:08.763 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T11:02:08.763 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ create_image cluster2 mirror image_50 128M 2023-10-11T11:02:08.764 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:02:08.764 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T11:02:08.764 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:02:08.764 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T11:02:08.765 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_50 2023-10-11T11:02:08.765 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T11:02:08.765 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local size=128 2023-10-11T11:02:08.765 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n 128M ']' 2023-10-11T11:02:08.766 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ size=128M 2023-10-11T11:02:08.766 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T11:02:08.766 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_50 2023-10-11T11:02:08.807 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' image = image ']' 2023-10-11T11:02:08.807 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ enable_mirror cluster2 mirror image_50 snapshot 2023-10-11T11:02:08.808 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:02:08.808 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:02:08.808 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_50 2023-10-11T11:02:08.809 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local mode=snapshot 2023-10-11T11:02:08.809 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_50 snapshot 2023-10-11T11:02:09.566 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.7398856062512656) > weight (0.25) 2023-10-11T11:02:09.566 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.4400204975569846) > weight (0.25) 2023-10-11T11:02:09.566 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.8061238746110477) > weight (0.25) 2023-10-11T11:02:09.569 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.3 2023-10-11T11:02:09.569 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Sent signal 15 2023-10-11T11:02:09.570 INFO:tasks.rbd_mirror_thrash:waiting for 1 secs before reviving daemons 2023-10-11T11:02:09.571 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi111.stderr:2023-10-11T11:02:09.570+0000 7f3aba536700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 (PID: 111141) UID: 1000 2023-10-11T11:02:09.738 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:Mirroring enabled 2023-10-11T11:02:09.752 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 info mirror/image_50 2023-10-11T11:02:09.791 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:rbd image 'image_50': 2023-10-11T11:02:09.791 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: size 128 MiB in 32 objects 2023-10-11T11:02:09.792 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: order 22 (4 MiB objects) 2023-10-11T11:02:09.792 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: snapshot_count: 1 2023-10-11T11:02:09.792 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: id: 143c923326c7 2023-10-11T11:02:09.792 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: block_name_prefix: rbd_data.143c923326c7 2023-10-11T11:02:09.793 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: format: 2 2023-10-11T11:02:09.793 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-10-11T11:02:09.793 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: op_features: 2023-10-11T11:02:09.793 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: flags: 2023-10-11T11:02:09.794 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: create_timestamp: Wed Oct 11 11:02:08 2023 2023-10-11T11:02:09.794 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: access_timestamp: Wed Oct 11 11:02:08 2023 2023-10-11T11:02:09.794 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: modify_timestamp: Wed Oct 11 11:02:08 2023 2023-10-11T11:02:09.794 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring state: enabled 2023-10-11T11:02:09.795 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring mode: snapshot 2023-10-11T11:02:09.795 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring global id: cd60b48a-16a6-4a4e-8bc0-70e977b76bac 2023-10-11T11:02:09.795 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring primary: true 2023-10-11T11:02:09.797 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n '' ']' 2023-10-11T11:02:09.797 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ stress_write_image cluster2 mirror image_50 2023-10-11T11:02:09.797 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:02:09.798 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:02:09.798 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_50 2023-10-11T11:02:09.798 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-10-11T11:02:09.798 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local duration=7 2023-10-11T11:02:09.799 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ set +e 2023-10-11T11:02:09.799 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_50 --debug-rbd=20 --debug-journaler=20 2023-10-11T11:02:10.065 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:100 IOs, 40663040 bytes 2023-10-11T11:02:10.291 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:200 IOs, 103884800 bytes 2023-10-11T11:02:10.422 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:300 IOs, 115533824 bytes 2023-10-11T11:02:10.571 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.3 2023-10-11T11:02:10.572 INFO:teuthology.orchestra.run:waiting for 600 2023-10-11T11:02:10.572 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.3 2023-10-11T11:02:10.572 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Restarting daemon 2023-10-11T11:02:10.572 DEBUG:teuthology.orchestra.run.smithi111:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 2023-10-11T11:02:10.575 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Started 2023-10-11T11:02:10.575 INFO:tasks.rbd_mirror_thrash:waiting for 72 secs before thrashing 2023-10-11T11:02:10.645 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:400 IOs, 172775424 bytes 2023-10-11T11:02:10.796 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:500 IOs, 184640512 bytes 2023-10-11T11:02:11.025 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:600 IOs, 243571712 bytes 2023-10-11T11:02:11.218 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:700 IOs, 290823168 bytes 2023-10-11T11:02:11.440 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:800 IOs, 345939968 bytes 2023-10-11T11:02:11.699 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:900 IOs, 414934016 bytes 2023-10-11T11:02:11.886 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1000 IOs, 452427776 bytes 2023-10-11T11:02:12.069 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1100 IOs, 486865920 bytes 2023-10-11T11:02:12.304 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1200 IOs, 554589184 bytes 2023-10-11T11:02:12.629 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1300 IOs, 655951872 bytes 2023-10-11T11:02:12.853 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1400 IOs, 687202304 bytes 2023-10-11T11:02:13.072 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1500 IOs, 741782528 bytes 2023-10-11T11:02:13.339 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1600 IOs, 817824768 bytes 2023-10-11T11:02:13.496 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1700 IOs, 845420544 bytes 2023-10-11T11:02:13.756 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1800 IOs, 932983808 bytes 2023-10-11T11:02:14.001 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1900 IOs, 979483648 bytes 2023-10-11T11:02:14.194 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2000 IOs, 1027474432 bytes 2023-10-11T11:02:14.435 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2100 IOs, 1095863296 bytes 2023-10-11T11:02:14.657 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2200 IOs, 1153640448 bytes 2023-10-11T11:02:14.850 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2300 IOs, 1216933888 bytes 2023-10-11T11:02:15.124 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2400 IOs, 1288152064 bytes 2023-10-11T11:02:15.328 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2500 IOs, 1338528768 bytes 2023-10-11T11:02:15.510 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2600 IOs, 1381525504 bytes 2023-10-11T11:02:15.703 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2700 IOs, 1431324672 bytes 2023-10-11T11:02:15.833 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2800 IOs, 1442195456 bytes 2023-10-11T11:02:15.997 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2900 IOs, 1473486848 bytes 2023-10-11T11:02:16.161 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3000 IOs, 1509841920 bytes 2023-10-11T11:02:16.359 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3100 IOs, 1549546496 bytes 2023-10-11T11:02:16.665 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3200 IOs, 1647512576 bytes 2023-10-11T11:02:16.801 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ error_code=124 2023-10-11T11:02:16.801 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ set -e 2023-10-11T11:02:16.801 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' 124 -eq 124 ']' 2023-10-11T11:02:16.802 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:02:16.802 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_pool_images cluster2 mirror 50 2023-10-11T11:02:16.802 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:02:16.802 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:02:16.802 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image_count=50 2023-10-11T11:02:16.802 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T11:02:16.803 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local count 2023-10-11T11:02:16.803 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local last_count=0 2023-10-11T11:02:16.803 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T11:02:16.803 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ seq 1 40 2023-10-11T11:02:16.804 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in `seq 1 40` 2023-10-11T11:02:16.804 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 1 -ne 1 2023-10-11T11:02:16.804 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ rbd --cluster cluster2 -p mirror mirror pool status 2023-10-11T11:02:16.804 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ grep 'images: ' 2023-10-11T11:02:16.805 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ cut '-d ' -f 2 2023-10-11T11:02:16.838 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd: failed to query services: (13) Permission denied 2023-10-11T11:02:16.841 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ count=50 2023-10-11T11:02:16.842 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 50 = 50 2023-10-11T11:02:16.842 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:02:16.842 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_pool_healthy cluster2 mirror 2023-10-11T11:02:16.842 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:02:16.842 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:02:16.843 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T11:02:16.843 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local state 2023-10-11T11:02:16.843 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ seq 1 40 2023-10-11T11:02:16.844 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in `seq 1 40` 2023-10-11T11:02:16.844 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 1 -ne 1 2023-10-11T11:02:16.844 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ rbd --cluster cluster2 -p mirror mirror pool status 2023-10-11T11:02:16.844 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ grep 'image health:' 2023-10-11T11:02:16.844 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ cut '-d ' -f 3 2023-10-11T11:02:16.876 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd: failed to query services: (13) Permission denied 2023-10-11T11:02:16.880 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ state=OK 2023-10-11T11:02:16.880 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test OK = ERROR 2023-10-11T11:02:16.880 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test OK = OK 2023-10-11T11:02:16.881 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:02:16.881 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_pool_images cluster1 mirror 50 2023-10-11T11:02:16.881 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:02:16.881 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:02:16.881 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image_count=50 2023-10-11T11:02:16.881 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T11:02:16.882 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local count 2023-10-11T11:02:16.882 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local last_count=0 2023-10-11T11:02:16.882 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T11:02:16.882 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ seq 1 40 2023-10-11T11:02:16.882 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in `seq 1 40` 2023-10-11T11:02:16.882 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 1 -ne 1 2023-10-11T11:02:16.883 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ rbd --cluster cluster1 -p mirror mirror pool status 2023-10-11T11:02:16.883 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ grep 'images: ' 2023-10-11T11:02:16.883 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ cut '-d ' -f 2 2023-10-11T11:02:16.918 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd: failed to query services: (13) Permission denied 2023-10-11T11:02:16.921 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ count=50 2023-10-11T11:02:16.921 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 50 = 50 2023-10-11T11:02:16.922 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:02:16.922 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_pool_healthy cluster1 mirror 2023-10-11T11:02:16.922 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:02:16.922 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:02:16.922 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T11:02:16.923 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local state 2023-10-11T11:02:16.923 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ seq 1 40 2023-10-11T11:02:16.924 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in `seq 1 40` 2023-10-11T11:02:16.924 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 1 -ne 1 2023-10-11T11:02:16.924 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ rbd --cluster cluster1 -p mirror mirror pool status 2023-10-11T11:02:16.924 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ grep 'image health:' 2023-10-11T11:02:16.924 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ cut '-d ' -f 3 2023-10-11T11:02:16.958 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd: failed to query services: (13) Permission denied 2023-10-11T11:02:16.961 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ state=WARNING 2023-10-11T11:02:16.962 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test WARNING = ERROR 2023-10-11T11:02:16.962 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test WARNING = OK 2023-10-11T11:02:16.962 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in `seq 1 40` 2023-10-11T11:02:16.962 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 2 -ne 1 2023-10-11T11:02:16.962 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 30 2023-10-11T11:02:46.964 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ rbd --cluster cluster1 -p mirror mirror pool status 2023-10-11T11:02:46.964 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ grep 'image health:' 2023-10-11T11:02:46.965 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ cut '-d ' -f 3 2023-10-11T11:02:47.006 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd: failed to query services: (13) Permission denied 2023-10-11T11:02:47.009 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ state=OK 2023-10-11T11:02:47.010 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test OK = ERROR 2023-10-11T11:02:47.010 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test OK = OK 2023-10-11T11:02:47.010 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:02:47.010 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ testlog 'TEST: compare many images' 2023-10-11T11:02:47.011 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ tee -a /tmp/tmp.nGgfBEp0YO/rbd-mirror.test.log 2023-10-11T11:02:47.011 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ date '+%F %T' 2023-10-11T11:02:47.012 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ echo 2023-10-11 11:02:47 TEST: compare many images 2023-10-11T11:02:47.013 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:2023-10-11 11:02:47 TEST: compare many images 2023-10-11T11:02:47.013 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ seq 1 50 2023-10-11T11:02:47.013 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T11:02:47.013 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_1 2023-10-11T11:02:47.013 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ create_snap cluster2 mirror image_1 snap 2023-10-11T11:02:47.014 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:02:47.014 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:02:47.014 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_1 2023-10-11T11:02:47.014 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap 2023-10-11T11:02:47.014 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 -p mirror snap create image_1@snap --debug-rbd=20 --debug-journaler=20 2023-10-11T11:02:48.472 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_replay_started cluster1 mirror image_1 2023-10-11T11:02:48.472 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:02:48.472 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:02:48.473 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_1 2023-10-11T11:02:48.473 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_replay_state cluster1 mirror image_1 started 2023-10-11T11:02:48.473 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:02:48.473 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:02:48.474 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_1 2023-10-11T11:02:48.474 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local state=started 2023-10-11T11:02:48.474 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T11:02:48.475 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-10-11T11:02:48.475 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 1 2023-10-11T11:02:49.474 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test_image_replay_state cluster1 mirror image_1 started 2023-10-11T11:02:49.474 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:02:49.474 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:02:49.475 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_1 2023-10-11T11:02:49.475 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local test_state=started 2023-10-11T11:02:49.475 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local status_result 2023-10-11T11:02:49.475 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_state=stopped 2023-10-11T11:02:49.476 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_1 2023-10-11T11:02:49.476 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_instance=cluster1 2023-10-11T11:02:49.476 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T11:02:49.477 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:02:49.477 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance=cluster1 2023-10-11T11:02:49.477 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local loop_instance 2023-10-11T11:02:49.477 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ grep -i state 2023-10-11T11:02:49.478 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-10-11T11:02:49.478 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sleep 0 2023-10-11T11:02:49.482 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-10-11T11:02:49.482 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ seq 0 3 2023-10-11T11:02:49.483 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-10-11T11:02:49.483 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_1 2023-10-11T11:02:49.483 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1:0 2023-10-11T11:02:49.483 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T11:02:49.484 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance 2023-10-11T11:02:49.484 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-10-11T11:02:49.484 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local val=cluster1:0 2023-10-11T11:02:49.484 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_var_name=cluster 2023-10-11T11:02:49.485 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance_var_name=instance 2023-10-11T11:02:49.485 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ cluster=cluster1 2023-10-11T11:02:49.485 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ instance=0 2023-10-11T11:02:49.485 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' 0 = cluster1:0 ']' 2023-10-11T11:02:49.486 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval cluster=cluster1 2023-10-11T11:02:49.486 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:02:49.486 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval instance=0 2023-10-11T11:02:49.487 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=0 2023-10-11T11:02:49.487 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-10-11T11:02:49.487 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local local_cluster=cluster1:0 2023-10-11T11:02:49.487 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster=cluster1 2023-10-11T11:02:49.488 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance 2023-10-11T11:02:49.488 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-10-11T11:02:49.488 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local val=cluster1:0 2023-10-11T11:02:49.488 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster_var_name=local_cluster 2023-10-11T11:02:49.489 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance_var_name=instance 2023-10-11T11:02:49.489 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:02:49.489 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=0 2023-10-11T11:02:49.489 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ '[' 0 = cluster1:0 ']' 2023-10-11T11:02:49.490 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval local_cluster=cluster1 2023-10-11T11:02:49.490 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ local_cluster=cluster1 2023-10-11T11:02:49.490 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval instance=0 2023-10-11T11:02:49.490 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ instance=0 2023-10-11T11:02:49.491 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-10-11T11:02:49.500 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T11:02:49.501 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T11:02:49.501 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T11:02:49.501 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_1 2023-10-11T11:02:49.605 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:Invalid command: unused arguments: ['mirror/image_1'] 2023-10-11T11:02:49.606 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd mirror status : get status for rbd mirror 2023-10-11T11:02:49.606 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:admin_socket: invalid command 2023-10-11T11:02:49.609 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-10-11T11:02:49.610 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_1 2023-10-11T11:02:49.610 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1:1 2023-10-11T11:02:49.610 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T11:02:49.611 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance 2023-10-11T11:02:49.611 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-10-11T11:02:49.611 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local val=cluster1:1 2023-10-11T11:02:49.611 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_var_name=cluster 2023-10-11T11:02:49.612 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance_var_name=instance 2023-10-11T11:02:49.612 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ cluster=cluster1 2023-10-11T11:02:49.612 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ instance=1 2023-10-11T11:02:49.612 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' 1 = cluster1:1 ']' 2023-10-11T11:02:49.613 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval cluster=cluster1 2023-10-11T11:02:49.613 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:02:49.613 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval instance=1 2023-10-11T11:02:49.613 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=1 2023-10-11T11:02:49.614 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-10-11T11:02:49.614 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local local_cluster=cluster1:1 2023-10-11T11:02:49.614 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster=cluster1 2023-10-11T11:02:49.615 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance 2023-10-11T11:02:49.615 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-10-11T11:02:49.615 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local val=cluster1:1 2023-10-11T11:02:49.615 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster_var_name=local_cluster 2023-10-11T11:02:49.616 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance_var_name=instance 2023-10-11T11:02:49.616 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:02:49.616 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=1 2023-10-11T11:02:49.616 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ '[' 1 = cluster1:1 ']' 2023-10-11T11:02:49.617 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval local_cluster=cluster1 2023-10-11T11:02:49.617 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ local_cluster=cluster1 2023-10-11T11:02:49.617 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval instance=1 2023-10-11T11:02:49.617 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ instance=1 2023-10-11T11:02:49.618 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-10-11T11:02:49.628 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T11:02:49.628 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T11:02:49.628 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T11:02:49.629 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_1 2023-10-11T11:02:49.740 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:Invalid command: unused arguments: ['mirror/image_1'] 2023-10-11T11:02:49.741 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd mirror status : get status for rbd mirror 2023-10-11T11:02:49.741 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:admin_socket: invalid command 2023-10-11T11:02:49.744 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-10-11T11:02:49.745 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_1 2023-10-11T11:02:49.745 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1:2 2023-10-11T11:02:49.745 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T11:02:49.746 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance 2023-10-11T11:02:49.746 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-10-11T11:02:49.746 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local val=cluster1:2 2023-10-11T11:02:49.747 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_var_name=cluster 2023-10-11T11:02:49.747 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance_var_name=instance 2023-10-11T11:02:49.747 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ cluster=cluster1 2023-10-11T11:02:49.747 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ instance=2 2023-10-11T11:02:49.748 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' 2 = cluster1:2 ']' 2023-10-11T11:02:49.748 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval cluster=cluster1 2023-10-11T11:02:49.748 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:02:49.748 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval instance=2 2023-10-11T11:02:49.749 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=2 2023-10-11T11:02:49.749 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-10-11T11:02:49.749 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local local_cluster=cluster1:2 2023-10-11T11:02:49.749 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster=cluster1 2023-10-11T11:02:49.750 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance 2023-10-11T11:02:49.750 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-10-11T11:02:49.750 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local val=cluster1:2 2023-10-11T11:02:49.750 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster_var_name=local_cluster 2023-10-11T11:02:49.751 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance_var_name=instance 2023-10-11T11:02:49.751 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:02:49.751 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=2 2023-10-11T11:02:49.751 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ '[' 2 = cluster1:2 ']' 2023-10-11T11:02:49.752 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval local_cluster=cluster1 2023-10-11T11:02:49.752 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ local_cluster=cluster1 2023-10-11T11:02:49.752 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval instance=2 2023-10-11T11:02:49.753 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ instance=2 2023-10-11T11:02:49.753 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-10-11T11:02:49.763 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-10-11T11:02:49.763 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-10-11T11:02:49.763 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-10-11T11:02:49.764 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_1 2023-10-11T11:02:49.914 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ return 0 2023-10-11T11:02:49.915 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ status_result=' "state": "Replaying"' 2023-10-11T11:02:49.915 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ echo ' "state": "Replaying"' 2023-10-11T11:02:49.915 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ grep -i Replaying 2023-10-11T11:02:49.916 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: "state": "Replaying" 2023-10-11T11:02:49.919 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ current_state=started 2023-10-11T11:02:49.919 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test started = started 2023-10-11T11:02:49.920 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:02:49.920 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_1 2023-10-11T11:02:49.920 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local local_cluster=cluster1 2023-10-11T11:02:49.920 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:02:49.921 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:02:49.921 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_1 2023-10-11T11:02:49.921 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' snapshot = journal ']' 2023-10-11T11:02:49.921 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' snapshot = snapshot ']' 2023-10-11T11:02:49.922 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_1 2023-10-11T11:02:49.922 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local local_cluster=cluster1 2023-10-11T11:02:49.922 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:02:49.923 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:02:49.923 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_1 2023-10-11T11:02:49.923 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ mkfname cluster2-mirror-image_1.status 2023-10-11T11:02:49.923 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ echo cluster2-mirror-image_1.status 2023-10-11T11:02:49.924 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -e 's|[/ ]|_|g' 2023-10-11T11:02:49.924 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local status_log=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_1.status 2023-10-11T11:02:49.924 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ mkfname cluster1-mirror-image_1.status 2023-10-11T11:02:49.924 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ echo cluster1-mirror-image_1.status 2023-10-11T11:02:49.925 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -e 's|[/ ]|_|g' 2023-10-11T11:02:49.925 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local local_status_log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_1.status 2023-10-11T11:02:49.925 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ mirror_image_snapshot cluster2 mirror image_1 2023-10-11T11:02:49.925 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:02:49.926 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:02:49.926 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_1 2023-10-11T11:02:49.926 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_1 2023-10-11T11:02:50.467 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:Snapshot ID: 119 2023-10-11T11:02:50.478 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_1 /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_1.status 2023-10-11T11:02:50.478 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:02:50.479 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:02:50.479 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_1 2023-10-11T11:02:50.479 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_1.status 2023-10-11T11:02:50.479 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_1 --format xml 2023-10-11T11:02:50.480 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:02:50.521 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-10-11T11:02:50.523 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snapshot_id=119 2023-10-11T11:02:50.523 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T11:02:50.523 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T11:02:50.524 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.2 2023-10-11T11:02:50.724 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_1 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_1.status 2023-10-11T11:02:50.724 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:02:50.724 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:02:50.725 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_1 2023-10-11T11:02:50.725 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_1.status 2023-10-11T11:02:50.725 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_1 --format xml 2023-10-11T11:02:50.725 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:02:50.771 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T11:02:50.773 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id=68 2023-10-11T11:02:50.773 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 119 = 68 2023-10-11T11:02:50.773 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T11:02:50.773 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.4 2023-10-11T11:02:51.174 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_1 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_1.status 2023-10-11T11:02:51.174 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:02:51.174 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:02:51.175 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_1 2023-10-11T11:02:51.175 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_1.status 2023-10-11T11:02:51.175 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_1 --format xml 2023-10-11T11:02:51.175 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:02:51.221 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T11:02:51.221 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T11:02:51.222 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:-:1.1: Document is empty 2023-10-11T11:02:51.222 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: 2023-10-11T11:02:51.223 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:^ 2023-10-11T11:02:51.223 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id= 2023-10-11T11:02:51.224 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 119 = '' 2023-10-11T11:02:51.224 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T11:02:51.224 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.8 2023-10-11T11:02:52.025 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_1 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_1.status 2023-10-11T11:02:52.025 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:02:52.025 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:02:52.025 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_1 2023-10-11T11:02:52.025 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_1.status 2023-10-11T11:02:52.026 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_1 --format xml 2023-10-11T11:02:52.026 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:02:52.092 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T11:02:52.092 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T11:02:52.093 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:-:1.1: Document is empty 2023-10-11T11:02:52.093 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: 2023-10-11T11:02:52.094 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:^ 2023-10-11T11:02:52.094 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id= 2023-10-11T11:02:52.095 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 119 = '' 2023-10-11T11:02:52.095 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T11:02:52.095 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 1.6 2023-10-11T11:02:53.695 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_1 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_1.status 2023-10-11T11:02:53.696 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:02:53.696 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:02:53.696 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_1 2023-10-11T11:02:53.696 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_1.status 2023-10-11T11:02:53.696 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_1 --format xml 2023-10-11T11:02:53.697 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:02:53.747 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T11:02:53.749 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id=119 2023-10-11T11:02:53.750 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 119 = 119 2023-10-11T11:02:53.750 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:02:53.750 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_snap_present cluster1 mirror image_1 snap 2023-10-11T11:02:53.750 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:02:53.750 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:02:53.751 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_1 2023-10-11T11:02:53.751 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap 2023-10-11T11:02:53.751 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T11:02:53.751 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-10-11T11:02:53.751 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 1 2023-10-11T11:02:54.750 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 info mirror/image_1@snap 2023-10-11T11:02:54.797 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:rbd image 'image_1': 2023-10-11T11:02:54.797 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: size 128 MiB in 32 objects 2023-10-11T11:02:54.797 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: order 22 (4 MiB objects) 2023-10-11T11:02:54.798 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: snapshot_count: 2 2023-10-11T11:02:54.798 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: id: 10af4ca68d34 2023-10-11T11:02:54.798 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: block_name_prefix: rbd_data.10af4ca68d34 2023-10-11T11:02:54.798 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: format: 2 2023-10-11T11:02:54.799 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-10-11T11:02:54.799 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: op_features: 2023-10-11T11:02:54.799 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: flags: 2023-10-11T11:02:54.799 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: create_timestamp: Wed Oct 11 10:54:11 2023 2023-10-11T11:02:54.800 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: access_timestamp: Wed Oct 11 10:54:11 2023 2023-10-11T11:02:54.800 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: modify_timestamp: Wed Oct 11 10:54:11 2023 2023-10-11T11:02:54.800 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: protected: False 2023-10-11T11:02:54.800 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring state: enabled 2023-10-11T11:02:54.800 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring mode: snapshot 2023-10-11T11:02:54.801 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring global id: eeb747e2-80b4-485a-95e6-a07d67abb191 2023-10-11T11:02:54.801 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring primary: false 2023-10-11T11:02:54.802 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:02:54.802 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ compare_image_snaps mirror image_1 snap 2023-10-11T11:02:54.803 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:02:54.803 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_1 2023-10-11T11:02:54.803 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap 2023-10-11T11:02:54.803 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local ret=0 2023-10-11T11:02:54.804 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local rmt_export=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_1.export 2023-10-11T11:02:54.804 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local loc_export=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_1.export 2023-10-11T11:02:54.804 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rm -f /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_1.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_1.export 2023-10-11T11:02:54.804 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 -p mirror export image_1@snap /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_1.export 2023-10-11T11:02:54.996 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-10-11T11:02:55.005 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 -p mirror export image_1@snap /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_1.export 2023-10-11T11:02:55.226 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-10-11T11:02:55.233 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ cmp /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_1.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_1.export 2023-10-11T11:02:55.290 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rm -f /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_1.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_1.export 2023-10-11T11:02:55.327 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:02:55.327 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T11:02:55.328 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_2 2023-10-11T11:02:55.328 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ create_snap cluster2 mirror image_2 snap 2023-10-11T11:02:55.328 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:02:55.328 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:02:55.328 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_2 2023-10-11T11:02:55.329 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap 2023-10-11T11:02:55.329 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 -p mirror snap create image_2@snap --debug-rbd=20 --debug-journaler=20 2023-10-11T11:02:57.385 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_replay_started cluster1 mirror image_2 2023-10-11T11:02:57.385 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:02:57.385 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:02:57.386 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_2 2023-10-11T11:02:57.386 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_replay_state cluster1 mirror image_2 started 2023-10-11T11:02:57.386 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:02:57.386 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:02:57.387 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_2 2023-10-11T11:02:57.387 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local state=started 2023-10-11T11:02:57.387 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T11:02:57.387 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-10-11T11:02:57.387 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 1 2023-10-11T11:02:58.387 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test_image_replay_state cluster1 mirror image_2 started 2023-10-11T11:02:58.388 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:02:58.388 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:02:58.388 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_2 2023-10-11T11:02:58.389 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local test_state=started 2023-10-11T11:02:58.389 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local status_result 2023-10-11T11:02:58.389 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_state=stopped 2023-10-11T11:02:58.389 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_2 2023-10-11T11:02:58.390 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_instance=cluster1 2023-10-11T11:02:58.390 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T11:02:58.390 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:02:58.391 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ grep -i state 2023-10-11T11:02:58.391 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance=cluster1 2023-10-11T11:02:58.391 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local loop_instance 2023-10-11T11:02:58.391 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-10-11T11:02:58.392 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sleep 0 2023-10-11T11:02:58.392 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-10-11T11:02:58.395 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ seq 0 3 2023-10-11T11:02:58.396 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-10-11T11:02:58.396 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_2 2023-10-11T11:02:58.396 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1:0 2023-10-11T11:02:58.396 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T11:02:58.397 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance 2023-10-11T11:02:58.397 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-10-11T11:02:58.397 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local val=cluster1:0 2023-10-11T11:02:58.397 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_var_name=cluster 2023-10-11T11:02:58.398 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance_var_name=instance 2023-10-11T11:02:58.398 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ cluster=cluster1 2023-10-11T11:02:58.398 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ instance=0 2023-10-11T11:02:58.399 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' 0 = cluster1:0 ']' 2023-10-11T11:02:58.399 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval cluster=cluster1 2023-10-11T11:02:58.399 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:02:58.399 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval instance=0 2023-10-11T11:02:58.400 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=0 2023-10-11T11:02:58.400 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-10-11T11:02:58.400 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local local_cluster=cluster1:0 2023-10-11T11:02:58.400 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster=cluster1 2023-10-11T11:02:58.401 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance 2023-10-11T11:02:58.401 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-10-11T11:02:58.401 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local val=cluster1:0 2023-10-11T11:02:58.401 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster_var_name=local_cluster 2023-10-11T11:02:58.402 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance_var_name=instance 2023-10-11T11:02:58.402 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:02:58.402 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=0 2023-10-11T11:02:58.402 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ '[' 0 = cluster1:0 ']' 2023-10-11T11:02:58.403 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval local_cluster=cluster1 2023-10-11T11:02:58.403 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ local_cluster=cluster1 2023-10-11T11:02:58.403 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval instance=0 2023-10-11T11:02:58.403 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ instance=0 2023-10-11T11:02:58.404 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-10-11T11:02:58.414 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T11:02:58.414 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T11:02:58.414 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T11:02:58.415 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_2 2023-10-11T11:02:58.514 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:Invalid command: unused arguments: ['mirror/image_2'] 2023-10-11T11:02:58.514 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd mirror status : get status for rbd mirror 2023-10-11T11:02:58.514 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:admin_socket: invalid command 2023-10-11T11:02:58.518 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-10-11T11:02:58.518 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_2 2023-10-11T11:02:58.519 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1:1 2023-10-11T11:02:58.519 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T11:02:58.519 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance 2023-10-11T11:02:58.519 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-10-11T11:02:58.520 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local val=cluster1:1 2023-10-11T11:02:58.520 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_var_name=cluster 2023-10-11T11:02:58.520 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance_var_name=instance 2023-10-11T11:02:58.520 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ cluster=cluster1 2023-10-11T11:02:58.521 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ instance=1 2023-10-11T11:02:58.521 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' 1 = cluster1:1 ']' 2023-10-11T11:02:58.521 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval cluster=cluster1 2023-10-11T11:02:58.521 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:02:58.522 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval instance=1 2023-10-11T11:02:58.522 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=1 2023-10-11T11:02:58.522 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-10-11T11:02:58.522 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local local_cluster=cluster1:1 2023-10-11T11:02:58.523 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster=cluster1 2023-10-11T11:02:58.523 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance 2023-10-11T11:02:58.523 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-10-11T11:02:58.524 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local val=cluster1:1 2023-10-11T11:02:58.524 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster_var_name=local_cluster 2023-10-11T11:02:58.524 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance_var_name=instance 2023-10-11T11:02:58.524 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:02:58.525 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=1 2023-10-11T11:02:58.525 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ '[' 1 = cluster1:1 ']' 2023-10-11T11:02:58.525 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval local_cluster=cluster1 2023-10-11T11:02:58.525 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ local_cluster=cluster1 2023-10-11T11:02:58.526 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval instance=1 2023-10-11T11:02:58.526 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ instance=1 2023-10-11T11:02:58.526 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-10-11T11:02:58.536 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T11:02:58.536 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T11:02:58.537 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T11:02:58.537 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_2 2023-10-11T11:02:58.631 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:Invalid command: unused arguments: ['mirror/image_2'] 2023-10-11T11:02:58.631 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd mirror status : get status for rbd mirror 2023-10-11T11:02:58.631 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:admin_socket: invalid command 2023-10-11T11:02:58.635 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-10-11T11:02:58.635 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_2 2023-10-11T11:02:58.636 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1:2 2023-10-11T11:02:58.636 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T11:02:58.636 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance 2023-10-11T11:02:58.636 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-10-11T11:02:58.637 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local val=cluster1:2 2023-10-11T11:02:58.637 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_var_name=cluster 2023-10-11T11:02:58.637 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance_var_name=instance 2023-10-11T11:02:58.637 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ cluster=cluster1 2023-10-11T11:02:58.638 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ instance=2 2023-10-11T11:02:58.638 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' 2 = cluster1:2 ']' 2023-10-11T11:02:58.638 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval cluster=cluster1 2023-10-11T11:02:58.639 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:02:58.639 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval instance=2 2023-10-11T11:02:58.639 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=2 2023-10-11T11:02:58.639 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-10-11T11:02:58.640 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local local_cluster=cluster1:2 2023-10-11T11:02:58.640 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster=cluster1 2023-10-11T11:02:58.640 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance 2023-10-11T11:02:58.640 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-10-11T11:02:58.641 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local val=cluster1:2 2023-10-11T11:02:58.641 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster_var_name=local_cluster 2023-10-11T11:02:58.641 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance_var_name=instance 2023-10-11T11:02:58.641 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:02:58.642 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=2 2023-10-11T11:02:58.642 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ '[' 2 = cluster1:2 ']' 2023-10-11T11:02:58.642 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval local_cluster=cluster1 2023-10-11T11:02:58.642 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ local_cluster=cluster1 2023-10-11T11:02:58.643 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval instance=2 2023-10-11T11:02:58.643 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ instance=2 2023-10-11T11:02:58.643 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-10-11T11:02:58.652 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-10-11T11:02:58.653 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-10-11T11:02:58.653 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-10-11T11:02:58.653 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_2 2023-10-11T11:02:58.801 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ return 0 2023-10-11T11:02:58.801 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ status_result=' "state": "Replaying"' 2023-10-11T11:02:58.801 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ echo ' "state": "Replaying"' 2023-10-11T11:02:58.802 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ grep -i Replaying 2023-10-11T11:02:58.802 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: "state": "Replaying" 2023-10-11T11:02:58.806 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ current_state=started 2023-10-11T11:02:58.806 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test started = started 2023-10-11T11:02:58.806 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:02:58.807 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_2 2023-10-11T11:02:58.807 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local local_cluster=cluster1 2023-10-11T11:02:58.807 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:02:58.807 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:02:58.808 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_2 2023-10-11T11:02:58.808 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' snapshot = journal ']' 2023-10-11T11:02:58.808 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' snapshot = snapshot ']' 2023-10-11T11:02:58.808 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_2 2023-10-11T11:02:58.809 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local local_cluster=cluster1 2023-10-11T11:02:58.809 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:02:58.809 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:02:58.810 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_2 2023-10-11T11:02:58.810 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ mkfname cluster2-mirror-image_2.status 2023-10-11T11:02:58.810 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ echo cluster2-mirror-image_2.status 2023-10-11T11:02:58.810 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -e 's|[/ ]|_|g' 2023-10-11T11:02:58.811 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local status_log=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_2.status 2023-10-11T11:02:58.811 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ mkfname cluster1-mirror-image_2.status 2023-10-11T11:02:58.811 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ echo cluster1-mirror-image_2.status 2023-10-11T11:02:58.811 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -e 's|[/ ]|_|g' 2023-10-11T11:02:58.812 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local local_status_log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_2.status 2023-10-11T11:02:58.812 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ mirror_image_snapshot cluster2 mirror image_2 2023-10-11T11:02:58.812 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:02:58.812 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:02:58.813 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_2 2023-10-11T11:02:58.813 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_2 2023-10-11T11:02:59.389 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:Snapshot ID: 121 2023-10-11T11:02:59.400 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_2 /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_2.status 2023-10-11T11:02:59.400 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:02:59.401 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:02:59.401 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_2 2023-10-11T11:02:59.401 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_2.status 2023-10-11T11:02:59.402 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_2 --format xml 2023-10-11T11:02:59.402 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:02:59.442 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-10-11T11:02:59.444 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snapshot_id=121 2023-10-11T11:02:59.445 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T11:02:59.445 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T11:02:59.445 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.2 2023-10-11T11:02:59.646 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_2 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_2.status 2023-10-11T11:02:59.646 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:02:59.647 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:02:59.647 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_2 2023-10-11T11:02:59.647 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_2.status 2023-10-11T11:02:59.647 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_2 --format xml 2023-10-11T11:02:59.648 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:02:59.693 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T11:02:59.695 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id=69 2023-10-11T11:02:59.695 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 121 = 69 2023-10-11T11:02:59.695 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T11:02:59.696 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.4 2023-10-11T11:03:00.096 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_2 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_2.status 2023-10-11T11:03:00.097 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:03:00.097 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:03:00.097 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_2 2023-10-11T11:03:00.097 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_2.status 2023-10-11T11:03:00.097 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_2 --format xml 2023-10-11T11:03:00.098 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:03:00.144 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T11:03:00.144 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T11:03:00.146 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:-:1.1: Document is empty 2023-10-11T11:03:00.146 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: 2023-10-11T11:03:00.146 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:^ 2023-10-11T11:03:00.146 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id= 2023-10-11T11:03:00.147 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 121 = '' 2023-10-11T11:03:00.147 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T11:03:00.147 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.8 2023-10-11T11:03:00.947 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_2 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_2.status 2023-10-11T11:03:00.947 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:03:00.948 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:03:00.948 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_2 2023-10-11T11:03:00.948 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_2.status 2023-10-11T11:03:00.948 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_2 --format xml 2023-10-11T11:03:00.948 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:03:01.005 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T11:03:01.005 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T11:03:01.007 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:-:1.1: Document is empty 2023-10-11T11:03:01.008 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: 2023-10-11T11:03:01.008 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:^ 2023-10-11T11:03:01.008 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id= 2023-10-11T11:03:01.009 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 121 = '' 2023-10-11T11:03:01.009 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T11:03:01.009 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 1.6 2023-10-11T11:03:02.610 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_2 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_2.status 2023-10-11T11:03:02.610 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:03:02.610 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:03:02.611 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_2 2023-10-11T11:03:02.611 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_2.status 2023-10-11T11:03:02.611 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_2 --format xml 2023-10-11T11:03:02.611 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:03:02.665 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T11:03:02.667 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id=121 2023-10-11T11:03:02.667 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 121 = 121 2023-10-11T11:03:02.668 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:03:02.668 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_snap_present cluster1 mirror image_2 snap 2023-10-11T11:03:02.668 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:03:02.668 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:03:02.668 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_2 2023-10-11T11:03:02.669 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap 2023-10-11T11:03:02.669 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T11:03:02.669 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-10-11T11:03:02.669 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 1 2023-10-11T11:03:03.668 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 info mirror/image_2@snap 2023-10-11T11:03:03.713 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:rbd image 'image_2': 2023-10-11T11:03:03.714 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: size 128 MiB in 32 objects 2023-10-11T11:03:03.714 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: order 22 (4 MiB objects) 2023-10-11T11:03:03.714 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: snapshot_count: 2 2023-10-11T11:03:03.714 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: id: 10af9b95c4ce 2023-10-11T11:03:03.715 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: block_name_prefix: rbd_data.10af9b95c4ce 2023-10-11T11:03:03.715 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: format: 2 2023-10-11T11:03:03.715 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-10-11T11:03:03.715 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: op_features: 2023-10-11T11:03:03.716 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: flags: 2023-10-11T11:03:03.716 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: create_timestamp: Wed Oct 11 10:54:20 2023 2023-10-11T11:03:03.716 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: access_timestamp: Wed Oct 11 10:54:20 2023 2023-10-11T11:03:03.716 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: modify_timestamp: Wed Oct 11 10:54:20 2023 2023-10-11T11:03:03.717 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: protected: False 2023-10-11T11:03:03.717 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring state: enabled 2023-10-11T11:03:03.717 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring mode: snapshot 2023-10-11T11:03:03.718 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring global id: 115d7fef-7d84-4b5a-aaf4-a5ee31ecb254 2023-10-11T11:03:03.718 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring primary: false 2023-10-11T11:03:03.719 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:03:03.719 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ compare_image_snaps mirror image_2 snap 2023-10-11T11:03:03.720 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:03:03.720 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_2 2023-10-11T11:03:03.720 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap 2023-10-11T11:03:03.720 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local ret=0 2023-10-11T11:03:03.721 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local rmt_export=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_2.export 2023-10-11T11:03:03.721 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local loc_export=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_2.export 2023-10-11T11:03:03.721 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rm -f /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_2.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_2.export 2023-10-11T11:03:03.722 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 -p mirror export image_2@snap /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_2.export 2023-10-11T11:03:03.905 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-10-11T11:03:03.913 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 -p mirror export image_2@snap /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_2.export 2023-10-11T11:03:04.152 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-10-11T11:03:04.160 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ cmp /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_2.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_2.export 2023-10-11T11:03:04.215 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rm -f /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_2.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_2.export 2023-10-11T11:03:04.253 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:03:04.253 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T11:03:04.253 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_3 2023-10-11T11:03:04.253 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ create_snap cluster2 mirror image_3 snap 2023-10-11T11:03:04.254 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:03:04.254 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:03:04.254 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_3 2023-10-11T11:03:04.255 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap 2023-10-11T11:03:04.255 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 -p mirror snap create image_3@snap --debug-rbd=20 --debug-journaler=20 2023-10-11T11:03:05.461 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_replay_started cluster1 mirror image_3 2023-10-11T11:03:05.461 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:03:05.462 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:03:05.462 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_3 2023-10-11T11:03:05.462 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_replay_state cluster1 mirror image_3 started 2023-10-11T11:03:05.462 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:03:05.462 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:03:05.463 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_3 2023-10-11T11:03:05.463 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local state=started 2023-10-11T11:03:05.463 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T11:03:05.463 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-10-11T11:03:05.463 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 1 2023-10-11T11:03:06.463 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test_image_replay_state cluster1 mirror image_3 started 2023-10-11T11:03:06.463 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:03:06.463 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:03:06.463 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_3 2023-10-11T11:03:06.464 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local test_state=started 2023-10-11T11:03:06.464 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local status_result 2023-10-11T11:03:06.464 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_state=stopped 2023-10-11T11:03:06.464 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_3 2023-10-11T11:03:06.464 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_instance=cluster1 2023-10-11T11:03:06.464 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T11:03:06.465 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ grep -i state 2023-10-11T11:03:06.465 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:03:06.465 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance=cluster1 2023-10-11T11:03:06.465 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local loop_instance 2023-10-11T11:03:06.465 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-10-11T11:03:06.466 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sleep 0 2023-10-11T11:03:06.467 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-10-11T11:03:06.467 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ seq 0 3 2023-10-11T11:03:06.468 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-10-11T11:03:06.468 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_3 2023-10-11T11:03:06.468 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1:0 2023-10-11T11:03:06.468 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T11:03:06.468 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance 2023-10-11T11:03:06.469 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-10-11T11:03:06.469 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local val=cluster1:0 2023-10-11T11:03:06.469 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_var_name=cluster 2023-10-11T11:03:06.469 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance_var_name=instance 2023-10-11T11:03:06.469 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ cluster=cluster1 2023-10-11T11:03:06.469 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ instance=0 2023-10-11T11:03:06.470 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' 0 = cluster1:0 ']' 2023-10-11T11:03:06.470 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval cluster=cluster1 2023-10-11T11:03:06.470 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:03:06.470 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval instance=0 2023-10-11T11:03:06.470 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=0 2023-10-11T11:03:06.471 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-10-11T11:03:06.471 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local local_cluster=cluster1:0 2023-10-11T11:03:06.472 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster=cluster1 2023-10-11T11:03:06.472 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance 2023-10-11T11:03:06.473 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-10-11T11:03:06.473 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local val=cluster1:0 2023-10-11T11:03:06.473 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster_var_name=local_cluster 2023-10-11T11:03:06.473 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance_var_name=instance 2023-10-11T11:03:06.473 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:03:06.473 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=0 2023-10-11T11:03:06.474 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ '[' 0 = cluster1:0 ']' 2023-10-11T11:03:06.474 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval local_cluster=cluster1 2023-10-11T11:03:06.474 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ local_cluster=cluster1 2023-10-11T11:03:06.474 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval instance=0 2023-10-11T11:03:06.474 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ instance=0 2023-10-11T11:03:06.474 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-10-11T11:03:06.489 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T11:03:06.489 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T11:03:06.489 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T11:03:06.489 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_3 2023-10-11T11:03:06.589 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:Invalid command: unused arguments: ['mirror/image_3'] 2023-10-11T11:03:06.589 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd mirror status : get status for rbd mirror 2023-10-11T11:03:06.589 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:admin_socket: invalid command 2023-10-11T11:03:06.592 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-10-11T11:03:06.592 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_3 2023-10-11T11:03:06.592 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1:1 2023-10-11T11:03:06.593 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T11:03:06.593 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance 2023-10-11T11:03:06.593 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-10-11T11:03:06.593 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local val=cluster1:1 2023-10-11T11:03:06.593 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_var_name=cluster 2023-10-11T11:03:06.593 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance_var_name=instance 2023-10-11T11:03:06.594 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ cluster=cluster1 2023-10-11T11:03:06.594 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ instance=1 2023-10-11T11:03:06.594 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' 1 = cluster1:1 ']' 2023-10-11T11:03:06.594 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval cluster=cluster1 2023-10-11T11:03:06.594 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:03:06.594 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval instance=1 2023-10-11T11:03:06.595 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=1 2023-10-11T11:03:06.595 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-10-11T11:03:06.595 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local local_cluster=cluster1:1 2023-10-11T11:03:06.595 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster=cluster1 2023-10-11T11:03:06.595 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance 2023-10-11T11:03:06.596 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-10-11T11:03:06.596 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local val=cluster1:1 2023-10-11T11:03:06.596 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster_var_name=local_cluster 2023-10-11T11:03:06.596 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance_var_name=instance 2023-10-11T11:03:06.596 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:03:06.596 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=1 2023-10-11T11:03:06.597 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ '[' 1 = cluster1:1 ']' 2023-10-11T11:03:06.597 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval local_cluster=cluster1 2023-10-11T11:03:06.597 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ local_cluster=cluster1 2023-10-11T11:03:06.597 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval instance=1 2023-10-11T11:03:06.597 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ instance=1 2023-10-11T11:03:06.597 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-10-11T11:03:06.611 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T11:03:06.611 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T11:03:06.611 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T11:03:06.611 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_3 2023-10-11T11:03:06.705 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:Invalid command: unused arguments: ['mirror/image_3'] 2023-10-11T11:03:06.706 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd mirror status : get status for rbd mirror 2023-10-11T11:03:06.706 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:admin_socket: invalid command 2023-10-11T11:03:06.708 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-10-11T11:03:06.709 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_3 2023-10-11T11:03:06.709 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1:2 2023-10-11T11:03:06.709 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T11:03:06.709 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance 2023-10-11T11:03:06.709 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-10-11T11:03:06.709 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local val=cluster1:2 2023-10-11T11:03:06.710 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_var_name=cluster 2023-10-11T11:03:06.710 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance_var_name=instance 2023-10-11T11:03:06.710 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ cluster=cluster1 2023-10-11T11:03:06.710 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ instance=2 2023-10-11T11:03:06.710 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' 2 = cluster1:2 ']' 2023-10-11T11:03:06.710 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval cluster=cluster1 2023-10-11T11:03:06.711 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:03:06.711 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval instance=2 2023-10-11T11:03:06.711 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=2 2023-10-11T11:03:06.711 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-10-11T11:03:06.711 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local local_cluster=cluster1:2 2023-10-11T11:03:06.712 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster=cluster1 2023-10-11T11:03:06.712 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance 2023-10-11T11:03:06.712 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-10-11T11:03:06.712 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local val=cluster1:2 2023-10-11T11:03:06.712 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster_var_name=local_cluster 2023-10-11T11:03:06.712 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance_var_name=instance 2023-10-11T11:03:06.713 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:03:06.713 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=2 2023-10-11T11:03:06.713 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ '[' 2 = cluster1:2 ']' 2023-10-11T11:03:06.713 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval local_cluster=cluster1 2023-10-11T11:03:06.713 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ local_cluster=cluster1 2023-10-11T11:03:06.713 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval instance=2 2023-10-11T11:03:06.714 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ instance=2 2023-10-11T11:03:06.714 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-10-11T11:03:06.729 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-10-11T11:03:06.729 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-10-11T11:03:06.729 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-10-11T11:03:06.729 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_3 2023-10-11T11:03:06.877 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ return 0 2023-10-11T11:03:06.878 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ status_result=' "state": "Replaying"' 2023-10-11T11:03:06.878 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ echo ' "state": "Replaying"' 2023-10-11T11:03:06.878 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ grep -i Replaying 2023-10-11T11:03:06.879 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: "state": "Replaying" 2023-10-11T11:03:06.880 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ current_state=started 2023-10-11T11:03:06.881 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test started = started 2023-10-11T11:03:06.881 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:03:06.881 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_3 2023-10-11T11:03:06.881 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local local_cluster=cluster1 2023-10-11T11:03:06.881 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:03:06.882 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:03:06.882 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_3 2023-10-11T11:03:06.882 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' snapshot = journal ']' 2023-10-11T11:03:06.882 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' snapshot = snapshot ']' 2023-10-11T11:03:06.882 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_3 2023-10-11T11:03:06.882 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local local_cluster=cluster1 2023-10-11T11:03:06.883 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:03:06.883 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:03:06.883 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_3 2023-10-11T11:03:06.883 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ mkfname cluster2-mirror-image_3.status 2023-10-11T11:03:06.883 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ echo cluster2-mirror-image_3.status 2023-10-11T11:03:06.884 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -e 's|[/ ]|_|g' 2023-10-11T11:03:06.884 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local status_log=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_3.status 2023-10-11T11:03:06.885 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ mkfname cluster1-mirror-image_3.status 2023-10-11T11:03:06.885 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ echo cluster1-mirror-image_3.status 2023-10-11T11:03:06.885 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -e 's|[/ ]|_|g' 2023-10-11T11:03:06.885 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local local_status_log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_3.status 2023-10-11T11:03:06.886 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ mirror_image_snapshot cluster2 mirror image_3 2023-10-11T11:03:06.886 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:03:06.886 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:03:06.886 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_3 2023-10-11T11:03:06.886 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_3 2023-10-11T11:03:07.473 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:Snapshot ID: 123 2023-10-11T11:03:07.484 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_3 /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_3.status 2023-10-11T11:03:07.484 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:03:07.485 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:03:07.485 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_3 2023-10-11T11:03:07.485 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_3.status 2023-10-11T11:03:07.485 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_3 --format xml 2023-10-11T11:03:07.486 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:03:07.527 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-10-11T11:03:07.529 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snapshot_id=123 2023-10-11T11:03:07.529 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T11:03:07.529 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T11:03:07.529 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.2 2023-10-11T11:03:07.730 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_3 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_3.status 2023-10-11T11:03:07.730 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:03:07.730 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:03:07.731 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_3 2023-10-11T11:03:07.731 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_3.status 2023-10-11T11:03:07.731 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_3 --format xml 2023-10-11T11:03:07.731 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:03:07.776 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T11:03:07.776 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T11:03:07.777 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:-:1.1: Document is empty 2023-10-11T11:03:07.777 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: 2023-10-11T11:03:07.777 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:^ 2023-10-11T11:03:07.778 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id= 2023-10-11T11:03:07.779 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 123 = '' 2023-10-11T11:03:07.779 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T11:03:07.779 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.4 2023-10-11T11:03:08.179 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_3 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_3.status 2023-10-11T11:03:08.179 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:03:08.179 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:03:08.180 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_3 2023-10-11T11:03:08.180 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_3.status 2023-10-11T11:03:08.180 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_3 --format xml 2023-10-11T11:03:08.180 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:03:08.225 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T11:03:08.225 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T11:03:08.227 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:-:1.1: Document is empty 2023-10-11T11:03:08.227 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: 2023-10-11T11:03:08.227 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:^ 2023-10-11T11:03:08.227 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id= 2023-10-11T11:03:08.228 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 123 = '' 2023-10-11T11:03:08.228 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T11:03:08.228 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.8 2023-10-11T11:03:09.028 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_3 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_3.status 2023-10-11T11:03:09.028 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:03:09.029 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:03:09.029 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_3 2023-10-11T11:03:09.029 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_3.status 2023-10-11T11:03:09.029 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_3 --format xml 2023-10-11T11:03:09.030 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:03:09.073 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T11:03:09.076 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id=123 2023-10-11T11:03:09.076 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 123 = 123 2023-10-11T11:03:09.076 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:03:09.077 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_snap_present cluster1 mirror image_3 snap 2023-10-11T11:03:09.077 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:03:09.077 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:03:09.077 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_3 2023-10-11T11:03:09.077 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap 2023-10-11T11:03:09.078 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T11:03:09.078 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-10-11T11:03:09.078 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 1 2023-10-11T11:03:10.077 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 info mirror/image_3@snap 2023-10-11T11:03:10.122 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:rbd image 'image_3': 2023-10-11T11:03:10.122 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: size 128 MiB in 32 objects 2023-10-11T11:03:10.123 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: order 22 (4 MiB objects) 2023-10-11T11:03:10.123 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: snapshot_count: 2 2023-10-11T11:03:10.123 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: id: 10af84c4e848 2023-10-11T11:03:10.123 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: block_name_prefix: rbd_data.10af84c4e848 2023-10-11T11:03:10.124 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: format: 2 2023-10-11T11:03:10.124 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-10-11T11:03:10.124 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: op_features: 2023-10-11T11:03:10.124 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: flags: 2023-10-11T11:03:10.125 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: create_timestamp: Wed Oct 11 10:54:33 2023 2023-10-11T11:03:10.125 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: access_timestamp: Wed Oct 11 10:54:33 2023 2023-10-11T11:03:10.125 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: modify_timestamp: Wed Oct 11 10:54:33 2023 2023-10-11T11:03:10.125 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: protected: False 2023-10-11T11:03:10.126 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring state: enabled 2023-10-11T11:03:10.126 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring mode: snapshot 2023-10-11T11:03:10.126 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring global id: 665a8ec8-563e-4515-ba87-41db0aac57ac 2023-10-11T11:03:10.126 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring primary: false 2023-10-11T11:03:10.128 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:03:10.128 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ compare_image_snaps mirror image_3 snap 2023-10-11T11:03:10.128 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:03:10.129 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_3 2023-10-11T11:03:10.129 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap 2023-10-11T11:03:10.129 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local ret=0 2023-10-11T11:03:10.129 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local rmt_export=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_3.export 2023-10-11T11:03:10.130 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local loc_export=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_3.export 2023-10-11T11:03:10.130 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rm -f /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_3.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_3.export 2023-10-11T11:03:10.130 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 -p mirror export image_3@snap /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_3.export 2023-10-11T11:03:10.312 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-10-11T11:03:10.318 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 -p mirror export image_3@snap /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_3.export 2023-10-11T11:03:10.554 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-10-11T11:03:10.560 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ cmp /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_3.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_3.export 2023-10-11T11:03:10.615 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rm -f /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_3.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_3.export 2023-10-11T11:03:10.651 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:03:10.652 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T11:03:10.652 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_4 2023-10-11T11:03:10.652 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ create_snap cluster2 mirror image_4 snap 2023-10-11T11:03:10.652 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:03:10.652 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:03:10.653 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_4 2023-10-11T11:03:10.653 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap 2023-10-11T11:03:10.653 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 -p mirror snap create image_4@snap --debug-rbd=20 --debug-journaler=20 2023-10-11T11:03:12.386 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_replay_started cluster1 mirror image_4 2023-10-11T11:03:12.387 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:03:12.387 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:03:12.387 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_4 2023-10-11T11:03:12.388 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_replay_state cluster1 mirror image_4 started 2023-10-11T11:03:12.388 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:03:12.388 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:03:12.388 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_4 2023-10-11T11:03:12.388 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local state=started 2023-10-11T11:03:12.389 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T11:03:12.389 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-10-11T11:03:12.389 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 1 2023-10-11T11:03:13.387 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test_image_replay_state cluster1 mirror image_4 started 2023-10-11T11:03:13.388 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:03:13.388 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:03:13.388 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_4 2023-10-11T11:03:13.389 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local test_state=started 2023-10-11T11:03:13.389 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local status_result 2023-10-11T11:03:13.389 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_state=stopped 2023-10-11T11:03:13.393 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_4 2023-10-11T11:03:13.394 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_instance=cluster1 2023-10-11T11:03:13.394 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T11:03:13.394 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:03:13.395 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ grep -i state 2023-10-11T11:03:13.395 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance=cluster1 2023-10-11T11:03:13.395 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local loop_instance 2023-10-11T11:03:13.395 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-10-11T11:03:13.396 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sleep 0 2023-10-11T11:03:13.396 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-10-11T11:03:13.396 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ seq 0 3 2023-10-11T11:03:13.396 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-10-11T11:03:13.397 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_4 2023-10-11T11:03:13.397 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1:0 2023-10-11T11:03:13.397 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T11:03:13.397 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance 2023-10-11T11:03:13.398 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-10-11T11:03:13.398 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local val=cluster1:0 2023-10-11T11:03:13.398 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_var_name=cluster 2023-10-11T11:03:13.399 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance_var_name=instance 2023-10-11T11:03:13.399 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ cluster=cluster1 2023-10-11T11:03:13.399 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ instance=0 2023-10-11T11:03:13.399 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' 0 = cluster1:0 ']' 2023-10-11T11:03:13.399 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval cluster=cluster1 2023-10-11T11:03:13.400 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:03:13.400 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval instance=0 2023-10-11T11:03:13.400 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=0 2023-10-11T11:03:13.400 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-10-11T11:03:13.401 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local local_cluster=cluster1:0 2023-10-11T11:03:13.401 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster=cluster1 2023-10-11T11:03:13.401 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance 2023-10-11T11:03:13.401 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-10-11T11:03:13.402 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local val=cluster1:0 2023-10-11T11:03:13.402 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster_var_name=local_cluster 2023-10-11T11:03:13.402 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance_var_name=instance 2023-10-11T11:03:13.402 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:03:13.403 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=0 2023-10-11T11:03:13.403 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ '[' 0 = cluster1:0 ']' 2023-10-11T11:03:13.403 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval local_cluster=cluster1 2023-10-11T11:03:13.403 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ local_cluster=cluster1 2023-10-11T11:03:13.404 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval instance=0 2023-10-11T11:03:13.404 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ instance=0 2023-10-11T11:03:13.404 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-10-11T11:03:13.416 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T11:03:13.416 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T11:03:13.417 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T11:03:13.417 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_4 2023-10-11T11:03:13.516 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:Invalid command: unused arguments: ['mirror/image_4'] 2023-10-11T11:03:13.516 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd mirror status : get status for rbd mirror 2023-10-11T11:03:13.516 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:admin_socket: invalid command 2023-10-11T11:03:13.520 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-10-11T11:03:13.520 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_4 2023-10-11T11:03:13.520 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1:1 2023-10-11T11:03:13.520 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T11:03:13.521 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance 2023-10-11T11:03:13.521 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-10-11T11:03:13.521 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local val=cluster1:1 2023-10-11T11:03:13.521 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_var_name=cluster 2023-10-11T11:03:13.522 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance_var_name=instance 2023-10-11T11:03:13.522 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ cluster=cluster1 2023-10-11T11:03:13.522 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ instance=1 2023-10-11T11:03:13.522 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' 1 = cluster1:1 ']' 2023-10-11T11:03:13.523 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval cluster=cluster1 2023-10-11T11:03:13.523 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:03:13.523 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval instance=1 2023-10-11T11:03:13.523 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=1 2023-10-11T11:03:13.524 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-10-11T11:03:13.524 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local local_cluster=cluster1:1 2023-10-11T11:03:13.524 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster=cluster1 2023-10-11T11:03:13.524 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance 2023-10-11T11:03:13.525 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-10-11T11:03:13.525 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local val=cluster1:1 2023-10-11T11:03:13.525 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster_var_name=local_cluster 2023-10-11T11:03:13.525 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance_var_name=instance 2023-10-11T11:03:13.525 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:03:13.526 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=1 2023-10-11T11:03:13.526 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ '[' 1 = cluster1:1 ']' 2023-10-11T11:03:13.526 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval local_cluster=cluster1 2023-10-11T11:03:13.527 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ local_cluster=cluster1 2023-10-11T11:03:13.527 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval instance=1 2023-10-11T11:03:13.527 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ instance=1 2023-10-11T11:03:13.527 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-10-11T11:03:13.538 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T11:03:13.538 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T11:03:13.539 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T11:03:13.539 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_4 2023-10-11T11:03:13.631 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:Invalid command: unused arguments: ['mirror/image_4'] 2023-10-11T11:03:13.632 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd mirror status : get status for rbd mirror 2023-10-11T11:03:13.632 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:admin_socket: invalid command 2023-10-11T11:03:13.636 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-10-11T11:03:13.636 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_4 2023-10-11T11:03:13.636 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1:2 2023-10-11T11:03:13.636 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T11:03:13.637 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance 2023-10-11T11:03:13.637 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-10-11T11:03:13.637 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local val=cluster1:2 2023-10-11T11:03:13.637 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_var_name=cluster 2023-10-11T11:03:13.638 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance_var_name=instance 2023-10-11T11:03:13.638 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ cluster=cluster1 2023-10-11T11:03:13.638 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ instance=2 2023-10-11T11:03:13.638 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' 2 = cluster1:2 ']' 2023-10-11T11:03:13.639 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval cluster=cluster1 2023-10-11T11:03:13.639 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:03:13.639 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval instance=2 2023-10-11T11:03:13.639 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=2 2023-10-11T11:03:13.640 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-10-11T11:03:13.640 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local local_cluster=cluster1:2 2023-10-11T11:03:13.640 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster=cluster1 2023-10-11T11:03:13.640 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance 2023-10-11T11:03:13.641 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-10-11T11:03:13.641 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local val=cluster1:2 2023-10-11T11:03:13.641 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster_var_name=local_cluster 2023-10-11T11:03:13.641 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance_var_name=instance 2023-10-11T11:03:13.642 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:03:13.642 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=2 2023-10-11T11:03:13.642 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ '[' 2 = cluster1:2 ']' 2023-10-11T11:03:13.642 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval local_cluster=cluster1 2023-10-11T11:03:13.643 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ local_cluster=cluster1 2023-10-11T11:03:13.643 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval instance=2 2023-10-11T11:03:13.643 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ instance=2 2023-10-11T11:03:13.643 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-10-11T11:03:13.653 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-10-11T11:03:13.654 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-10-11T11:03:13.654 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-10-11T11:03:13.654 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_4 2023-10-11T11:03:13.806 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ return 0 2023-10-11T11:03:13.806 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ status_result=' "state": "Replaying"' 2023-10-11T11:03:13.807 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ echo ' "state": "Replaying"' 2023-10-11T11:03:13.807 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ grep -i Replaying 2023-10-11T11:03:13.808 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: "state": "Replaying" 2023-10-11T11:03:13.811 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ current_state=started 2023-10-11T11:03:13.811 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test started = started 2023-10-11T11:03:13.811 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:03:13.812 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_4 2023-10-11T11:03:13.812 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local local_cluster=cluster1 2023-10-11T11:03:13.812 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:03:13.812 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:03:13.813 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_4 2023-10-11T11:03:13.813 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' snapshot = journal ']' 2023-10-11T11:03:13.813 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' snapshot = snapshot ']' 2023-10-11T11:03:13.813 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_4 2023-10-11T11:03:13.814 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local local_cluster=cluster1 2023-10-11T11:03:13.814 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:03:13.814 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:03:13.814 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_4 2023-10-11T11:03:13.815 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ mkfname cluster2-mirror-image_4.status 2023-10-11T11:03:13.815 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ echo cluster2-mirror-image_4.status 2023-10-11T11:03:13.815 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -e 's|[/ ]|_|g' 2023-10-11T11:03:13.815 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local status_log=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_4.status 2023-10-11T11:03:13.816 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ mkfname cluster1-mirror-image_4.status 2023-10-11T11:03:13.816 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ echo cluster1-mirror-image_4.status 2023-10-11T11:03:13.816 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -e 's|[/ ]|_|g' 2023-10-11T11:03:13.817 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local local_status_log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_4.status 2023-10-11T11:03:13.817 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ mirror_image_snapshot cluster2 mirror image_4 2023-10-11T11:03:13.817 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:03:13.817 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:03:13.817 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_4 2023-10-11T11:03:13.818 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_4 2023-10-11T11:03:14.387 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:Snapshot ID: 125 2023-10-11T11:03:14.397 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_4 /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_4.status 2023-10-11T11:03:14.398 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:03:14.398 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:03:14.398 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_4 2023-10-11T11:03:14.398 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_4.status 2023-10-11T11:03:14.399 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_4 --format xml 2023-10-11T11:03:14.399 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:03:14.441 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-10-11T11:03:14.442 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snapshot_id=125 2023-10-11T11:03:14.443 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T11:03:14.443 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T11:03:14.443 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.2 2023-10-11T11:03:14.644 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_4 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_4.status 2023-10-11T11:03:14.644 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:03:14.644 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:03:14.645 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_4 2023-10-11T11:03:14.645 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_4.status 2023-10-11T11:03:14.645 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_4 --format xml 2023-10-11T11:03:14.645 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:03:14.690 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T11:03:14.690 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T11:03:14.692 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:-:1.1: Document is empty 2023-10-11T11:03:14.692 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: 2023-10-11T11:03:14.692 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:^ 2023-10-11T11:03:14.692 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id= 2023-10-11T11:03:14.693 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 125 = '' 2023-10-11T11:03:14.693 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T11:03:14.693 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.4 2023-10-11T11:03:15.093 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_4 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_4.status 2023-10-11T11:03:15.093 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:03:15.093 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:03:15.093 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_4 2023-10-11T11:03:15.094 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_4.status 2023-10-11T11:03:15.094 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_4 --format xml 2023-10-11T11:03:15.094 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:03:15.139 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T11:03:15.139 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T11:03:15.140 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:-:1.1: Document is empty 2023-10-11T11:03:15.141 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: 2023-10-11T11:03:15.141 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:^ 2023-10-11T11:03:15.141 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id= 2023-10-11T11:03:15.142 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 125 = '' 2023-10-11T11:03:15.142 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T11:03:15.142 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.8 2023-10-11T11:03:15.942 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_4 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_4.status 2023-10-11T11:03:15.942 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:03:15.943 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:03:15.943 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_4 2023-10-11T11:03:15.943 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_4.status 2023-10-11T11:03:15.943 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_4 --format xml 2023-10-11T11:03:15.943 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:03:16.065 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T11:03:16.068 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id=125 2023-10-11T11:03:16.068 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 125 = 125 2023-10-11T11:03:16.068 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:03:16.068 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_snap_present cluster1 mirror image_4 snap 2023-10-11T11:03:16.068 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:03:16.069 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:03:16.069 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_4 2023-10-11T11:03:16.069 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap 2023-10-11T11:03:16.069 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T11:03:16.069 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-10-11T11:03:16.070 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 1 2023-10-11T11:03:17.068 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 info mirror/image_4@snap 2023-10-11T11:03:17.115 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:rbd image 'image_4': 2023-10-11T11:03:17.115 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: size 128 MiB in 32 objects 2023-10-11T11:03:17.115 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: order 22 (4 MiB objects) 2023-10-11T11:03:17.116 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: snapshot_count: 2 2023-10-11T11:03:17.116 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: id: 10afbdbf1c25 2023-10-11T11:03:17.116 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: block_name_prefix: rbd_data.10afbdbf1c25 2023-10-11T11:03:17.116 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: format: 2 2023-10-11T11:03:17.116 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-10-11T11:03:17.117 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: op_features: 2023-10-11T11:03:17.117 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: flags: 2023-10-11T11:03:17.117 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: create_timestamp: Wed Oct 11 10:54:38 2023 2023-10-11T11:03:17.117 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: access_timestamp: Wed Oct 11 10:54:38 2023 2023-10-11T11:03:17.117 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: modify_timestamp: Wed Oct 11 10:54:38 2023 2023-10-11T11:03:17.117 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: protected: False 2023-10-11T11:03:17.118 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring state: enabled 2023-10-11T11:03:17.118 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring mode: snapshot 2023-10-11T11:03:17.118 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring global id: f031088c-5560-4c60-b77f-f68d8adaedd0 2023-10-11T11:03:17.118 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring primary: false 2023-10-11T11:03:17.119 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:03:17.119 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ compare_image_snaps mirror image_4 snap 2023-10-11T11:03:17.119 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:03:17.120 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_4 2023-10-11T11:03:17.120 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap 2023-10-11T11:03:17.120 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local ret=0 2023-10-11T11:03:17.120 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local rmt_export=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_4.export 2023-10-11T11:03:17.120 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local loc_export=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_4.export 2023-10-11T11:03:17.120 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rm -f /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_4.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_4.export 2023-10-11T11:03:17.121 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 -p mirror export image_4@snap /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_4.export 2023-10-11T11:03:17.320 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-10-11T11:03:17.327 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 -p mirror export image_4@snap /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_4.export 2023-10-11T11:03:17.566 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-10-11T11:03:17.573 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ cmp /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_4.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_4.export 2023-10-11T11:03:17.627 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rm -f /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_4.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_4.export 2023-10-11T11:03:17.664 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:03:17.664 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T11:03:17.664 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_5 2023-10-11T11:03:17.664 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ create_snap cluster2 mirror image_5 snap 2023-10-11T11:03:17.664 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:03:17.664 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:03:17.665 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_5 2023-10-11T11:03:17.665 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap 2023-10-11T11:03:17.665 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 -p mirror snap create image_5@snap --debug-rbd=20 --debug-journaler=20 2023-10-11T11:03:19.467 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_replay_started cluster1 mirror image_5 2023-10-11T11:03:19.467 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:03:19.467 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:03:19.467 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_5 2023-10-11T11:03:19.468 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_replay_state cluster1 mirror image_5 started 2023-10-11T11:03:19.468 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:03:19.468 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:03:19.468 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_5 2023-10-11T11:03:19.468 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local state=started 2023-10-11T11:03:19.468 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T11:03:19.469 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-10-11T11:03:19.469 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 1 2023-10-11T11:03:20.468 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test_image_replay_state cluster1 mirror image_5 started 2023-10-11T11:03:20.468 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:03:20.468 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:03:20.468 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_5 2023-10-11T11:03:20.469 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local test_state=started 2023-10-11T11:03:20.469 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local status_result 2023-10-11T11:03:20.469 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_state=stopped 2023-10-11T11:03:20.470 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_5 2023-10-11T11:03:20.470 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_instance=cluster1 2023-10-11T11:03:20.470 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ grep -i state 2023-10-11T11:03:20.471 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T11:03:20.471 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:03:20.471 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance=cluster1 2023-10-11T11:03:20.471 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local loop_instance 2023-10-11T11:03:20.471 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-10-11T11:03:20.472 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sleep 0 2023-10-11T11:03:20.472 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-10-11T11:03:20.472 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ seq 0 3 2023-10-11T11:03:20.473 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-10-11T11:03:20.474 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_5 2023-10-11T11:03:20.474 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1:0 2023-10-11T11:03:20.474 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T11:03:20.474 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance 2023-10-11T11:03:20.475 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-10-11T11:03:20.475 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local val=cluster1:0 2023-10-11T11:03:20.475 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_var_name=cluster 2023-10-11T11:03:20.475 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance_var_name=instance 2023-10-11T11:03:20.475 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ cluster=cluster1 2023-10-11T11:03:20.475 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ instance=0 2023-10-11T11:03:20.476 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' 0 = cluster1:0 ']' 2023-10-11T11:03:20.476 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval cluster=cluster1 2023-10-11T11:03:20.476 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:03:20.476 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval instance=0 2023-10-11T11:03:20.476 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=0 2023-10-11T11:03:20.477 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-10-11T11:03:20.477 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local local_cluster=cluster1:0 2023-10-11T11:03:20.477 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster=cluster1 2023-10-11T11:03:20.477 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance 2023-10-11T11:03:20.478 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-10-11T11:03:20.479 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local val=cluster1:0 2023-10-11T11:03:20.479 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster_var_name=local_cluster 2023-10-11T11:03:20.479 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance_var_name=instance 2023-10-11T11:03:20.479 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:03:20.479 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=0 2023-10-11T11:03:20.480 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ '[' 0 = cluster1:0 ']' 2023-10-11T11:03:20.480 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval local_cluster=cluster1 2023-10-11T11:03:20.480 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ local_cluster=cluster1 2023-10-11T11:03:20.480 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval instance=0 2023-10-11T11:03:20.480 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ instance=0 2023-10-11T11:03:20.481 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-10-11T11:03:20.495 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T11:03:20.496 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T11:03:20.496 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T11:03:20.496 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_5 2023-10-11T11:03:20.595 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:Invalid command: unused arguments: ['mirror/image_5'] 2023-10-11T11:03:20.595 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd mirror status : get status for rbd mirror 2023-10-11T11:03:20.595 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:admin_socket: invalid command 2023-10-11T11:03:20.598 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-10-11T11:03:20.599 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_5 2023-10-11T11:03:20.599 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1:1 2023-10-11T11:03:20.599 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T11:03:20.599 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance 2023-10-11T11:03:20.599 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-10-11T11:03:20.600 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local val=cluster1:1 2023-10-11T11:03:20.600 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_var_name=cluster 2023-10-11T11:03:20.600 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance_var_name=instance 2023-10-11T11:03:20.600 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ cluster=cluster1 2023-10-11T11:03:20.600 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ instance=1 2023-10-11T11:03:20.600 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' 1 = cluster1:1 ']' 2023-10-11T11:03:20.601 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval cluster=cluster1 2023-10-11T11:03:20.601 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:03:20.601 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval instance=1 2023-10-11T11:03:20.601 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=1 2023-10-11T11:03:20.601 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-10-11T11:03:20.602 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local local_cluster=cluster1:1 2023-10-11T11:03:20.602 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster=cluster1 2023-10-11T11:03:20.602 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance 2023-10-11T11:03:20.602 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-10-11T11:03:20.602 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local val=cluster1:1 2023-10-11T11:03:20.602 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster_var_name=local_cluster 2023-10-11T11:03:20.603 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance_var_name=instance 2023-10-11T11:03:20.603 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:03:20.603 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=1 2023-10-11T11:03:20.603 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ '[' 1 = cluster1:1 ']' 2023-10-11T11:03:20.603 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval local_cluster=cluster1 2023-10-11T11:03:20.604 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ local_cluster=cluster1 2023-10-11T11:03:20.604 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval instance=1 2023-10-11T11:03:20.604 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ instance=1 2023-10-11T11:03:20.604 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-10-11T11:03:20.617 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T11:03:20.617 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T11:03:20.617 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T11:03:20.617 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_5 2023-10-11T11:03:20.711 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:Invalid command: unused arguments: ['mirror/image_5'] 2023-10-11T11:03:20.712 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd mirror status : get status for rbd mirror 2023-10-11T11:03:20.712 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:admin_socket: invalid command 2023-10-11T11:03:20.715 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-10-11T11:03:20.715 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_5 2023-10-11T11:03:20.715 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1:2 2023-10-11T11:03:20.716 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T11:03:20.716 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance 2023-10-11T11:03:20.716 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-10-11T11:03:20.716 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local val=cluster1:2 2023-10-11T11:03:20.716 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_var_name=cluster 2023-10-11T11:03:20.717 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance_var_name=instance 2023-10-11T11:03:20.717 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ cluster=cluster1 2023-10-11T11:03:20.717 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ instance=2 2023-10-11T11:03:20.717 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' 2 = cluster1:2 ']' 2023-10-11T11:03:20.717 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval cluster=cluster1 2023-10-11T11:03:20.717 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:03:20.718 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval instance=2 2023-10-11T11:03:20.718 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=2 2023-10-11T11:03:20.718 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-10-11T11:03:20.718 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local local_cluster=cluster1:2 2023-10-11T11:03:20.718 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster=cluster1 2023-10-11T11:03:20.718 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance 2023-10-11T11:03:20.719 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-10-11T11:03:20.719 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local val=cluster1:2 2023-10-11T11:03:20.719 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster_var_name=local_cluster 2023-10-11T11:03:20.719 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance_var_name=instance 2023-10-11T11:03:20.719 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:03:20.720 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=2 2023-10-11T11:03:20.720 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ '[' 2 = cluster1:2 ']' 2023-10-11T11:03:20.720 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval local_cluster=cluster1 2023-10-11T11:03:20.720 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ local_cluster=cluster1 2023-10-11T11:03:20.720 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval instance=2 2023-10-11T11:03:20.720 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ instance=2 2023-10-11T11:03:20.721 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-10-11T11:03:20.735 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-10-11T11:03:20.735 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-10-11T11:03:20.735 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-10-11T11:03:20.735 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_5 2023-10-11T11:03:20.887 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ return 0 2023-10-11T11:03:20.888 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ status_result=' "state": "Replaying"' 2023-10-11T11:03:20.888 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ echo ' "state": "Replaying"' 2023-10-11T11:03:20.888 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ grep -i Replaying 2023-10-11T11:03:20.889 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: "state": "Replaying" 2023-10-11T11:03:20.890 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ current_state=started 2023-10-11T11:03:20.891 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test started = started 2023-10-11T11:03:20.891 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:03:20.891 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_5 2023-10-11T11:03:20.891 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local local_cluster=cluster1 2023-10-11T11:03:20.891 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:03:20.892 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:03:20.892 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_5 2023-10-11T11:03:20.892 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' snapshot = journal ']' 2023-10-11T11:03:20.892 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' snapshot = snapshot ']' 2023-10-11T11:03:20.892 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_5 2023-10-11T11:03:20.893 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local local_cluster=cluster1 2023-10-11T11:03:20.893 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:03:20.893 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:03:20.893 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_5 2023-10-11T11:03:20.893 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ mkfname cluster2-mirror-image_5.status 2023-10-11T11:03:20.893 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ echo cluster2-mirror-image_5.status 2023-10-11T11:03:20.894 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -e 's|[/ ]|_|g' 2023-10-11T11:03:20.895 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local status_log=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_5.status 2023-10-11T11:03:20.895 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ mkfname cluster1-mirror-image_5.status 2023-10-11T11:03:20.895 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ echo cluster1-mirror-image_5.status 2023-10-11T11:03:20.895 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -e 's|[/ ]|_|g' 2023-10-11T11:03:20.896 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local local_status_log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_5.status 2023-10-11T11:03:20.896 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ mirror_image_snapshot cluster2 mirror image_5 2023-10-11T11:03:20.896 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:03:20.896 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:03:20.896 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_5 2023-10-11T11:03:20.897 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_5 2023-10-11T11:03:21.471 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:Snapshot ID: 127 2023-10-11T11:03:21.482 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_5 /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_5.status 2023-10-11T11:03:21.482 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:03:21.482 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:03:21.482 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_5 2023-10-11T11:03:21.483 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_5.status 2023-10-11T11:03:21.483 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_5 --format xml 2023-10-11T11:03:21.483 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:03:21.525 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-10-11T11:03:21.527 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snapshot_id=127 2023-10-11T11:03:21.528 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T11:03:21.528 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T11:03:21.528 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.2 2023-10-11T11:03:21.729 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_5 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_5.status 2023-10-11T11:03:21.729 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:03:21.729 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:03:21.729 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_5 2023-10-11T11:03:21.729 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_5.status 2023-10-11T11:03:21.730 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_5 --format xml 2023-10-11T11:03:21.730 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:03:21.776 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T11:03:21.776 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T11:03:21.777 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:-:1.1: Document is empty 2023-10-11T11:03:21.778 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: 2023-10-11T11:03:21.778 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:^ 2023-10-11T11:03:21.778 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id= 2023-10-11T11:03:21.778 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 127 = '' 2023-10-11T11:03:21.778 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T11:03:21.779 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.4 2023-10-11T11:03:22.179 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_5 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_5.status 2023-10-11T11:03:22.179 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:03:22.179 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:03:22.180 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_5 2023-10-11T11:03:22.180 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_5.status 2023-10-11T11:03:22.180 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_5 --format xml 2023-10-11T11:03:22.180 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:03:22.228 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T11:03:22.228 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T11:03:22.229 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:-:1.1: Document is empty 2023-10-11T11:03:22.230 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: 2023-10-11T11:03:22.230 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:^ 2023-10-11T11:03:22.230 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id= 2023-10-11T11:03:22.231 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 127 = '' 2023-10-11T11:03:22.231 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T11:03:22.231 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.8 2023-10-11T11:03:22.576 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.6318408945138818) > weight (0.25) 2023-10-11T11:03:22.576 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.1 2023-10-11T11:03:22.577 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Sent signal 15 2023-10-11T11:03:22.577 INFO:tasks.rbd_mirror_thrash:waiting for 2 secs before reviving daemons 2023-10-11T11:03:22.578 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi111.stderr:2023-10-11T11:03:22.577+0000 7f6cb478f700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 115732) UID: 1000 2023-10-11T11:03:23.031 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_5 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_5.status 2023-10-11T11:03:23.032 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:03:23.032 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:03:23.032 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_5 2023-10-11T11:03:23.032 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_5.status 2023-10-11T11:03:23.032 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_5 --format xml 2023-10-11T11:03:23.033 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:03:23.077 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T11:03:23.079 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id=127 2023-10-11T11:03:23.079 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 127 = 127 2023-10-11T11:03:23.080 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:03:23.080 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_snap_present cluster1 mirror image_5 snap 2023-10-11T11:03:23.080 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:03:23.080 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:03:23.080 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_5 2023-10-11T11:03:23.081 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap 2023-10-11T11:03:23.081 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T11:03:23.081 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-10-11T11:03:23.081 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 1 2023-10-11T11:03:24.080 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 info mirror/image_5@snap 2023-10-11T11:03:24.128 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:rbd image 'image_5': 2023-10-11T11:03:24.128 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: size 128 MiB in 32 objects 2023-10-11T11:03:24.128 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: order 22 (4 MiB objects) 2023-10-11T11:03:24.129 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: snapshot_count: 2 2023-10-11T11:03:24.129 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: id: 10afe00c0add 2023-10-11T11:03:24.129 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: block_name_prefix: rbd_data.10afe00c0add 2023-10-11T11:03:24.129 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: format: 2 2023-10-11T11:03:24.130 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-10-11T11:03:24.130 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: op_features: 2023-10-11T11:03:24.130 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: flags: 2023-10-11T11:03:24.130 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: create_timestamp: Wed Oct 11 10:54:44 2023 2023-10-11T11:03:24.131 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: access_timestamp: Wed Oct 11 10:54:44 2023 2023-10-11T11:03:24.131 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: modify_timestamp: Wed Oct 11 10:54:44 2023 2023-10-11T11:03:24.131 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: protected: False 2023-10-11T11:03:24.131 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring state: enabled 2023-10-11T11:03:24.132 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring mode: snapshot 2023-10-11T11:03:24.132 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring global id: a9ff47ee-c486-4ecb-981d-63d0dafd14e5 2023-10-11T11:03:24.132 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring primary: false 2023-10-11T11:03:24.133 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:03:24.134 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ compare_image_snaps mirror image_5 snap 2023-10-11T11:03:24.134 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:03:24.134 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_5 2023-10-11T11:03:24.134 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap 2023-10-11T11:03:24.135 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local ret=0 2023-10-11T11:03:24.135 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local rmt_export=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_5.export 2023-10-11T11:03:24.135 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local loc_export=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_5.export 2023-10-11T11:03:24.135 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rm -f /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_5.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_5.export 2023-10-11T11:03:24.136 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 -p mirror export image_5@snap /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_5.export 2023-10-11T11:03:24.311 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-10-11T11:03:24.318 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 -p mirror export image_5@snap /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_5.export 2023-10-11T11:03:24.557 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.6946752945899007) > weight (0.25) 2023-10-11T11:03:24.557 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.8901339029988107) > weight (0.25) 2023-10-11T11:03:24.558 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.2 2023-10-11T11:03:24.558 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Sent signal 15 2023-10-11T11:03:24.559 INFO:tasks.rbd_mirror_thrash:waiting for 6 secs before reviving daemons 2023-10-11T11:03:24.559 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi111.stderr:2023-10-11T11:03:24.558+0000 7f6b64939700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 (PID: 108848) UID: 1000 2023-10-11T11:03:24.566 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-10-11T11:03:24.575 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ cmp /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_5.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_5.export 2023-10-11T11:03:24.578 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.1 2023-10-11T11:03:24.579 INFO:teuthology.orchestra.run:waiting for 600 2023-10-11T11:03:24.579 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.1 2023-10-11T11:03:24.580 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2023-10-11T11:03:24.580 DEBUG:teuthology.orchestra.run.smithi111:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2023-10-11T11:03:24.583 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2023-10-11T11:03:24.583 INFO:tasks.rbd_mirror_thrash:waiting for 80 secs before thrashing 2023-10-11T11:03:24.635 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rm -f /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_5.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_5.export 2023-10-11T11:03:24.674 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:03:24.674 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T11:03:24.675 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_6 2023-10-11T11:03:24.675 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ create_snap cluster2 mirror image_6 snap 2023-10-11T11:03:24.675 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:03:24.675 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:03:24.676 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_6 2023-10-11T11:03:24.676 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap 2023-10-11T11:03:24.676 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 -p mirror snap create image_6@snap --debug-rbd=20 --debug-journaler=20 2023-10-11T11:03:26.494 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_replay_started cluster1 mirror image_6 2023-10-11T11:03:26.494 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:03:26.495 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:03:26.495 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_6 2023-10-11T11:03:26.495 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_replay_state cluster1 mirror image_6 started 2023-10-11T11:03:26.495 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:03:26.495 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:03:26.495 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_6 2023-10-11T11:03:26.496 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local state=started 2023-10-11T11:03:26.496 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T11:03:26.496 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-10-11T11:03:26.496 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 1 2023-10-11T11:03:27.495 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test_image_replay_state cluster1 mirror image_6 started 2023-10-11T11:03:27.495 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:03:27.496 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:03:27.496 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_6 2023-10-11T11:03:27.496 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local test_state=started 2023-10-11T11:03:27.496 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local status_result 2023-10-11T11:03:27.496 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_state=stopped 2023-10-11T11:03:27.497 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_6 2023-10-11T11:03:27.498 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_instance=cluster1 2023-10-11T11:03:27.498 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T11:03:27.498 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ grep -i state 2023-10-11T11:03:27.498 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:03:27.499 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance=cluster1 2023-10-11T11:03:27.499 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local loop_instance 2023-10-11T11:03:27.499 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-10-11T11:03:27.499 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sleep 0 2023-10-11T11:03:27.499 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-10-11T11:03:27.499 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ seq 0 3 2023-10-11T11:03:27.502 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-10-11T11:03:27.502 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_6 2023-10-11T11:03:27.503 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1:0 2023-10-11T11:03:27.503 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T11:03:27.503 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance 2023-10-11T11:03:27.503 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-10-11T11:03:27.503 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local val=cluster1:0 2023-10-11T11:03:27.504 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_var_name=cluster 2023-10-11T11:03:27.504 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance_var_name=instance 2023-10-11T11:03:27.504 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ cluster=cluster1 2023-10-11T11:03:27.504 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ instance=0 2023-10-11T11:03:27.504 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' 0 = cluster1:0 ']' 2023-10-11T11:03:27.505 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval cluster=cluster1 2023-10-11T11:03:27.505 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:03:27.505 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval instance=0 2023-10-11T11:03:27.505 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=0 2023-10-11T11:03:27.505 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-10-11T11:03:27.506 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local local_cluster=cluster1:0 2023-10-11T11:03:27.506 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster=cluster1 2023-10-11T11:03:27.506 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance 2023-10-11T11:03:27.506 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-10-11T11:03:27.506 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local val=cluster1:0 2023-10-11T11:03:27.507 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster_var_name=local_cluster 2023-10-11T11:03:27.507 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance_var_name=instance 2023-10-11T11:03:27.507 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:03:27.507 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=0 2023-10-11T11:03:27.507 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ '[' 0 = cluster1:0 ']' 2023-10-11T11:03:27.507 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval local_cluster=cluster1 2023-10-11T11:03:27.508 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ local_cluster=cluster1 2023-10-11T11:03:27.508 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval instance=0 2023-10-11T11:03:27.508 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ instance=0 2023-10-11T11:03:27.508 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-10-11T11:03:27.522 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T11:03:27.522 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T11:03:27.522 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T11:03:27.522 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_6 2023-10-11T11:03:27.623 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:Invalid command: unused arguments: ['mirror/image_6'] 2023-10-11T11:03:27.623 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd mirror status : get status for rbd mirror 2023-10-11T11:03:27.623 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:admin_socket: invalid command 2023-10-11T11:03:27.626 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-10-11T11:03:27.627 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_6 2023-10-11T11:03:27.627 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1:1 2023-10-11T11:03:27.627 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T11:03:27.627 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance 2023-10-11T11:03:27.627 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-10-11T11:03:27.628 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local val=cluster1:1 2023-10-11T11:03:27.628 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_var_name=cluster 2023-10-11T11:03:27.628 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance_var_name=instance 2023-10-11T11:03:27.628 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ cluster=cluster1 2023-10-11T11:03:27.628 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ instance=1 2023-10-11T11:03:27.629 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' 1 = cluster1:1 ']' 2023-10-11T11:03:27.629 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval cluster=cluster1 2023-10-11T11:03:27.629 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:03:27.629 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval instance=1 2023-10-11T11:03:27.629 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=1 2023-10-11T11:03:27.629 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-10-11T11:03:27.630 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local local_cluster=cluster1:1 2023-10-11T11:03:27.630 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster=cluster1 2023-10-11T11:03:27.630 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance 2023-10-11T11:03:27.630 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-10-11T11:03:27.630 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local val=cluster1:1 2023-10-11T11:03:27.631 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster_var_name=local_cluster 2023-10-11T11:03:27.631 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance_var_name=instance 2023-10-11T11:03:27.631 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:03:27.631 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=1 2023-10-11T11:03:27.631 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ '[' 1 = cluster1:1 ']' 2023-10-11T11:03:27.632 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval local_cluster=cluster1 2023-10-11T11:03:27.632 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ local_cluster=cluster1 2023-10-11T11:03:27.632 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval instance=1 2023-10-11T11:03:27.632 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ instance=1 2023-10-11T11:03:27.632 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-10-11T11:03:27.646 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T11:03:27.646 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T11:03:27.646 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T11:03:27.647 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_6 2023-10-11T11:03:27.741 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:Invalid command: unused arguments: ['mirror/image_6'] 2023-10-11T11:03:27.742 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd mirror status : get status for rbd mirror 2023-10-11T11:03:27.742 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:admin_socket: invalid command 2023-10-11T11:03:27.745 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-10-11T11:03:27.745 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_6 2023-10-11T11:03:27.745 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1:2 2023-10-11T11:03:27.745 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T11:03:27.745 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance 2023-10-11T11:03:27.746 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-10-11T11:03:27.746 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local val=cluster1:2 2023-10-11T11:03:27.746 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_var_name=cluster 2023-10-11T11:03:27.746 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance_var_name=instance 2023-10-11T11:03:27.747 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ cluster=cluster1 2023-10-11T11:03:27.747 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ instance=2 2023-10-11T11:03:27.747 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' 2 = cluster1:2 ']' 2023-10-11T11:03:27.747 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval cluster=cluster1 2023-10-11T11:03:27.747 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:03:27.747 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval instance=2 2023-10-11T11:03:27.748 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=2 2023-10-11T11:03:27.748 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-10-11T11:03:27.748 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local local_cluster=cluster1:2 2023-10-11T11:03:27.748 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster=cluster1 2023-10-11T11:03:27.748 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance 2023-10-11T11:03:27.748 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-10-11T11:03:27.749 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local val=cluster1:2 2023-10-11T11:03:27.749 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster_var_name=local_cluster 2023-10-11T11:03:27.749 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance_var_name=instance 2023-10-11T11:03:27.749 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:03:27.749 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=2 2023-10-11T11:03:27.750 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ '[' 2 = cluster1:2 ']' 2023-10-11T11:03:27.750 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval local_cluster=cluster1 2023-10-11T11:03:27.750 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ local_cluster=cluster1 2023-10-11T11:03:27.750 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval instance=2 2023-10-11T11:03:27.750 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ instance=2 2023-10-11T11:03:27.751 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-10-11T11:03:27.764 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-10-11T11:03:27.764 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-10-11T11:03:27.764 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-10-11T11:03:27.764 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_6 2023-10-11T11:03:27.920 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ return 0 2023-10-11T11:03:27.921 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ status_result=' "state": "Replaying"' 2023-10-11T11:03:27.921 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ echo ' "state": "Replaying"' 2023-10-11T11:03:27.921 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ grep -i Replaying 2023-10-11T11:03:27.922 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: "state": "Replaying" 2023-10-11T11:03:27.924 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ current_state=started 2023-10-11T11:03:27.924 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test started = started 2023-10-11T11:03:27.924 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:03:27.924 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_6 2023-10-11T11:03:27.925 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local local_cluster=cluster1 2023-10-11T11:03:27.925 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:03:27.925 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:03:27.925 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_6 2023-10-11T11:03:27.925 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' snapshot = journal ']' 2023-10-11T11:03:27.926 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' snapshot = snapshot ']' 2023-10-11T11:03:27.926 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_6 2023-10-11T11:03:27.926 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local local_cluster=cluster1 2023-10-11T11:03:27.926 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:03:27.926 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:03:27.927 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_6 2023-10-11T11:03:27.927 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ mkfname cluster2-mirror-image_6.status 2023-10-11T11:03:27.927 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ echo cluster2-mirror-image_6.status 2023-10-11T11:03:27.927 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -e 's|[/ ]|_|g' 2023-10-11T11:03:27.927 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local status_log=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_6.status 2023-10-11T11:03:27.928 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ mkfname cluster1-mirror-image_6.status 2023-10-11T11:03:27.929 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ echo cluster1-mirror-image_6.status 2023-10-11T11:03:27.929 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -e 's|[/ ]|_|g' 2023-10-11T11:03:27.929 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local local_status_log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_6.status 2023-10-11T11:03:27.929 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ mirror_image_snapshot cluster2 mirror image_6 2023-10-11T11:03:27.930 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:03:27.930 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:03:27.930 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_6 2023-10-11T11:03:27.930 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_6 2023-10-11T11:03:28.494 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:Snapshot ID: 129 2023-10-11T11:03:28.505 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_6 /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_6.status 2023-10-11T11:03:28.506 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:03:28.506 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:03:28.506 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_6 2023-10-11T11:03:28.506 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_6.status 2023-10-11T11:03:28.507 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_6 --format xml 2023-10-11T11:03:28.507 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:03:28.548 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-10-11T11:03:28.551 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snapshot_id=129 2023-10-11T11:03:28.551 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T11:03:28.551 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T11:03:28.552 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.2 2023-10-11T11:03:28.751 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_6 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_6.status 2023-10-11T11:03:28.752 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:03:28.752 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:03:28.752 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_6 2023-10-11T11:03:28.752 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_6.status 2023-10-11T11:03:28.752 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_6 --format xml 2023-10-11T11:03:28.753 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:03:28.797 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T11:03:28.798 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T11:03:28.799 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:-:1.1: Document is empty 2023-10-11T11:03:28.799 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: 2023-10-11T11:03:28.799 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:^ 2023-10-11T11:03:28.800 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id= 2023-10-11T11:03:28.800 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 129 = '' 2023-10-11T11:03:28.800 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T11:03:28.800 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.4 2023-10-11T11:03:29.200 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_6 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_6.status 2023-10-11T11:03:29.201 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:03:29.201 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:03:29.201 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_6 2023-10-11T11:03:29.201 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_6.status 2023-10-11T11:03:29.202 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_6 --format xml 2023-10-11T11:03:29.202 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:03:29.247 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T11:03:29.248 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T11:03:29.249 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:-:1.1: Document is empty 2023-10-11T11:03:29.250 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: 2023-10-11T11:03:29.250 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:^ 2023-10-11T11:03:29.250 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id= 2023-10-11T11:03:29.250 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 129 = '' 2023-10-11T11:03:29.250 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T11:03:29.251 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.8 2023-10-11T11:03:30.051 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_6 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_6.status 2023-10-11T11:03:30.051 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:03:30.051 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:03:30.051 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_6 2023-10-11T11:03:30.052 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_6.status 2023-10-11T11:03:30.052 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_6 --format xml 2023-10-11T11:03:30.052 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:03:30.095 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T11:03:30.097 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id=129 2023-10-11T11:03:30.098 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 129 = 129 2023-10-11T11:03:30.098 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:03:30.098 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_snap_present cluster1 mirror image_6 snap 2023-10-11T11:03:30.098 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:03:30.098 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:03:30.098 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_6 2023-10-11T11:03:30.099 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap 2023-10-11T11:03:30.099 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T11:03:30.099 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-10-11T11:03:30.099 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 1 2023-10-11T11:03:30.559 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.2 2023-10-11T11:03:30.559 INFO:teuthology.orchestra.run:waiting for 600 2023-10-11T11:03:30.560 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.2 2023-10-11T11:03:30.560 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Restarting daemon 2023-10-11T11:03:30.560 DEBUG:teuthology.orchestra.run.smithi111:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 2023-10-11T11:03:30.563 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Started 2023-10-11T11:03:30.563 INFO:tasks.rbd_mirror_thrash:waiting for 99 secs before thrashing 2023-10-11T11:03:31.098 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 info mirror/image_6@snap 2023-10-11T11:03:31.140 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:rbd image 'image_6': 2023-10-11T11:03:31.140 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: size 128 MiB in 32 objects 2023-10-11T11:03:31.140 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: order 22 (4 MiB objects) 2023-10-11T11:03:31.141 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: snapshot_count: 2 2023-10-11T11:03:31.141 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: id: 10afe3422db0 2023-10-11T11:03:31.141 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: block_name_prefix: rbd_data.10afe3422db0 2023-10-11T11:03:31.141 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: format: 2 2023-10-11T11:03:31.141 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-10-11T11:03:31.142 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: op_features: 2023-10-11T11:03:31.142 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: flags: 2023-10-11T11:03:31.142 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: create_timestamp: Wed Oct 11 10:54:54 2023 2023-10-11T11:03:31.142 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: access_timestamp: Wed Oct 11 10:54:54 2023 2023-10-11T11:03:31.142 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: modify_timestamp: Wed Oct 11 10:54:54 2023 2023-10-11T11:03:31.142 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: protected: False 2023-10-11T11:03:31.143 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring state: enabled 2023-10-11T11:03:31.143 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring mode: snapshot 2023-10-11T11:03:31.143 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring global id: 7ce830b7-0256-4d8e-905d-503bebab5873 2023-10-11T11:03:31.143 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring primary: false 2023-10-11T11:03:31.144 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:03:31.144 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ compare_image_snaps mirror image_6 snap 2023-10-11T11:03:31.145 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:03:31.145 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_6 2023-10-11T11:03:31.145 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap 2023-10-11T11:03:31.145 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local ret=0 2023-10-11T11:03:31.145 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local rmt_export=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_6.export 2023-10-11T11:03:31.146 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local loc_export=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_6.export 2023-10-11T11:03:31.146 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rm -f /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_6.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_6.export 2023-10-11T11:03:31.146 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 -p mirror export image_6@snap /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_6.export 2023-10-11T11:03:31.334 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-10-11T11:03:31.341 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 -p mirror export image_6@snap /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_6.export 2023-10-11T11:03:31.591 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-10-11T11:03:31.598 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ cmp /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_6.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_6.export 2023-10-11T11:03:31.653 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rm -f /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_6.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_6.export 2023-10-11T11:03:31.689 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:03:31.689 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T11:03:31.690 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_7 2023-10-11T11:03:31.690 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ create_snap cluster2 mirror image_7 snap 2023-10-11T11:03:31.690 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:03:31.690 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:03:31.690 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_7 2023-10-11T11:03:31.691 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap 2023-10-11T11:03:31.691 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 -p mirror snap create image_7@snap --debug-rbd=20 --debug-journaler=20 2023-10-11T11:03:33.518 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_replay_started cluster1 mirror image_7 2023-10-11T11:03:33.518 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:03:33.519 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:03:33.519 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_7 2023-10-11T11:03:33.519 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_replay_state cluster1 mirror image_7 started 2023-10-11T11:03:33.519 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:03:33.519 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:03:33.519 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_7 2023-10-11T11:03:33.520 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local state=started 2023-10-11T11:03:33.520 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T11:03:33.520 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-10-11T11:03:33.520 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 1 2023-10-11T11:03:34.519 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test_image_replay_state cluster1 mirror image_7 started 2023-10-11T11:03:34.520 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:03:34.520 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:03:34.520 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_7 2023-10-11T11:03:34.520 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local test_state=started 2023-10-11T11:03:34.520 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local status_result 2023-10-11T11:03:34.521 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_state=stopped 2023-10-11T11:03:34.522 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_7 2023-10-11T11:03:34.522 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_instance=cluster1 2023-10-11T11:03:34.522 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ grep -i state 2023-10-11T11:03:34.523 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T11:03:34.523 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:03:34.523 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance=cluster1 2023-10-11T11:03:34.523 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local loop_instance 2023-10-11T11:03:34.523 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-10-11T11:03:34.523 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sleep 0 2023-10-11T11:03:34.524 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-10-11T11:03:34.524 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ seq 0 3 2023-10-11T11:03:34.526 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-10-11T11:03:34.526 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_7 2023-10-11T11:03:34.527 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1:0 2023-10-11T11:03:34.527 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T11:03:34.527 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance 2023-10-11T11:03:34.527 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-10-11T11:03:34.527 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local val=cluster1:0 2023-10-11T11:03:34.528 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_var_name=cluster 2023-10-11T11:03:34.528 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance_var_name=instance 2023-10-11T11:03:34.528 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ cluster=cluster1 2023-10-11T11:03:34.528 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ instance=0 2023-10-11T11:03:34.528 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' 0 = cluster1:0 ']' 2023-10-11T11:03:34.529 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval cluster=cluster1 2023-10-11T11:03:34.529 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:03:34.529 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval instance=0 2023-10-11T11:03:34.529 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=0 2023-10-11T11:03:34.529 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-10-11T11:03:34.529 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local local_cluster=cluster1:0 2023-10-11T11:03:34.530 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster=cluster1 2023-10-11T11:03:34.530 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance 2023-10-11T11:03:34.530 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-10-11T11:03:34.530 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local val=cluster1:0 2023-10-11T11:03:34.530 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster_var_name=local_cluster 2023-10-11T11:03:34.530 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance_var_name=instance 2023-10-11T11:03:34.531 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:03:34.531 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=0 2023-10-11T11:03:34.531 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ '[' 0 = cluster1:0 ']' 2023-10-11T11:03:34.531 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval local_cluster=cluster1 2023-10-11T11:03:34.531 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ local_cluster=cluster1 2023-10-11T11:03:34.532 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval instance=0 2023-10-11T11:03:34.532 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ instance=0 2023-10-11T11:03:34.532 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-10-11T11:03:34.547 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T11:03:34.547 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T11:03:34.548 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T11:03:34.548 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_7 2023-10-11T11:03:34.654 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:Invalid command: unused arguments: ['mirror/image_7'] 2023-10-11T11:03:34.655 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd mirror status : get status for rbd mirror 2023-10-11T11:03:34.655 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:admin_socket: invalid command 2023-10-11T11:03:34.657 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-10-11T11:03:34.658 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_7 2023-10-11T11:03:34.658 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1:1 2023-10-11T11:03:34.658 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T11:03:34.658 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance 2023-10-11T11:03:34.658 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-10-11T11:03:34.659 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local val=cluster1:1 2023-10-11T11:03:34.659 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_var_name=cluster 2023-10-11T11:03:34.659 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance_var_name=instance 2023-10-11T11:03:34.659 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ cluster=cluster1 2023-10-11T11:03:34.659 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ instance=1 2023-10-11T11:03:34.659 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' 1 = cluster1:1 ']' 2023-10-11T11:03:34.660 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval cluster=cluster1 2023-10-11T11:03:34.660 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:03:34.660 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval instance=1 2023-10-11T11:03:34.660 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=1 2023-10-11T11:03:34.660 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-10-11T11:03:34.660 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local local_cluster=cluster1:1 2023-10-11T11:03:34.661 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster=cluster1 2023-10-11T11:03:34.661 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance 2023-10-11T11:03:34.661 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-10-11T11:03:34.661 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local val=cluster1:1 2023-10-11T11:03:34.661 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster_var_name=local_cluster 2023-10-11T11:03:34.661 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance_var_name=instance 2023-10-11T11:03:34.662 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:03:34.662 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=1 2023-10-11T11:03:34.662 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ '[' 1 = cluster1:1 ']' 2023-10-11T11:03:34.662 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval local_cluster=cluster1 2023-10-11T11:03:34.662 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ local_cluster=cluster1 2023-10-11T11:03:34.663 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval instance=1 2023-10-11T11:03:34.663 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ instance=1 2023-10-11T11:03:34.663 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-10-11T11:03:34.677 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T11:03:34.677 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T11:03:34.678 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T11:03:34.678 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_7 2023-10-11T11:03:34.774 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:Invalid command: unused arguments: ['mirror/image_7'] 2023-10-11T11:03:34.774 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd mirror status : get status for rbd mirror 2023-10-11T11:03:34.774 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:admin_socket: invalid command 2023-10-11T11:03:34.777 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-10-11T11:03:34.777 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_7 2023-10-11T11:03:34.778 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1:2 2023-10-11T11:03:34.778 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T11:03:34.778 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance 2023-10-11T11:03:34.778 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-10-11T11:03:34.778 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local val=cluster1:2 2023-10-11T11:03:34.779 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_var_name=cluster 2023-10-11T11:03:34.779 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance_var_name=instance 2023-10-11T11:03:34.779 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ cluster=cluster1 2023-10-11T11:03:34.779 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ instance=2 2023-10-11T11:03:34.779 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' 2 = cluster1:2 ']' 2023-10-11T11:03:34.780 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval cluster=cluster1 2023-10-11T11:03:34.780 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:03:34.780 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval instance=2 2023-10-11T11:03:34.780 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=2 2023-10-11T11:03:34.780 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-10-11T11:03:34.780 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local local_cluster=cluster1:2 2023-10-11T11:03:34.781 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster=cluster1 2023-10-11T11:03:34.781 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance 2023-10-11T11:03:34.781 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-10-11T11:03:34.781 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local val=cluster1:2 2023-10-11T11:03:34.781 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster_var_name=local_cluster 2023-10-11T11:03:34.781 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance_var_name=instance 2023-10-11T11:03:34.782 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:03:34.782 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=2 2023-10-11T11:03:34.782 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ '[' 2 = cluster1:2 ']' 2023-10-11T11:03:34.782 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval local_cluster=cluster1 2023-10-11T11:03:34.782 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ local_cluster=cluster1 2023-10-11T11:03:34.782 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval instance=2 2023-10-11T11:03:34.783 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ instance=2 2023-10-11T11:03:34.783 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-10-11T11:03:34.796 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-10-11T11:03:34.796 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-10-11T11:03:34.796 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-10-11T11:03:34.797 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_7 2023-10-11T11:03:34.953 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ return 0 2023-10-11T11:03:34.953 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ status_result=' "state": "Replaying"' 2023-10-11T11:03:34.953 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ echo ' "state": "Replaying"' 2023-10-11T11:03:34.953 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ grep -i Replaying 2023-10-11T11:03:34.954 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: "state": "Replaying" 2023-10-11T11:03:34.956 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ current_state=started 2023-10-11T11:03:34.956 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test started = started 2023-10-11T11:03:34.956 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:03:34.957 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_7 2023-10-11T11:03:34.957 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local local_cluster=cluster1 2023-10-11T11:03:34.957 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:03:34.957 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:03:34.957 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_7 2023-10-11T11:03:34.958 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' snapshot = journal ']' 2023-10-11T11:03:34.958 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' snapshot = snapshot ']' 2023-10-11T11:03:34.958 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_7 2023-10-11T11:03:34.958 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local local_cluster=cluster1 2023-10-11T11:03:34.959 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:03:34.959 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:03:34.959 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_7 2023-10-11T11:03:34.959 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ mkfname cluster2-mirror-image_7.status 2023-10-11T11:03:34.959 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ echo cluster2-mirror-image_7.status 2023-10-11T11:03:34.960 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -e 's|[/ ]|_|g' 2023-10-11T11:03:34.960 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local status_log=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_7.status 2023-10-11T11:03:34.960 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ mkfname cluster1-mirror-image_7.status 2023-10-11T11:03:34.960 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ echo cluster1-mirror-image_7.status 2023-10-11T11:03:34.960 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -e 's|[/ ]|_|g' 2023-10-11T11:03:34.961 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local local_status_log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_7.status 2023-10-11T11:03:34.962 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ mirror_image_snapshot cluster2 mirror image_7 2023-10-11T11:03:34.962 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:03:34.962 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:03:34.962 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_7 2023-10-11T11:03:34.962 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_7 2023-10-11T11:03:35.515 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:Snapshot ID: 131 2023-10-11T11:03:35.527 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_7 /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_7.status 2023-10-11T11:03:35.527 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:03:35.527 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:03:35.527 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_7 2023-10-11T11:03:35.528 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_7.status 2023-10-11T11:03:35.528 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_7 --format xml 2023-10-11T11:03:35.528 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:03:35.570 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-10-11T11:03:35.572 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snapshot_id=131 2023-10-11T11:03:35.572 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T11:03:35.572 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T11:03:35.573 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.2 2023-10-11T11:03:35.773 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_7 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_7.status 2023-10-11T11:03:35.774 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:03:35.774 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:03:35.774 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_7 2023-10-11T11:03:35.774 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_7.status 2023-10-11T11:03:35.774 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_7 --format xml 2023-10-11T11:03:35.775 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:03:35.820 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T11:03:35.822 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id=74 2023-10-11T11:03:35.822 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 131 = 74 2023-10-11T11:03:35.822 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T11:03:35.823 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.4 2023-10-11T11:03:36.223 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_7 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_7.status 2023-10-11T11:03:36.224 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:03:36.224 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:03:36.224 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_7 2023-10-11T11:03:36.224 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_7.status 2023-10-11T11:03:36.225 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_7 --format xml 2023-10-11T11:03:36.225 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:03:36.270 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T11:03:36.270 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T11:03:36.272 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:-:1.1: Document is empty 2023-10-11T11:03:36.272 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: 2023-10-11T11:03:36.272 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:^ 2023-10-11T11:03:36.273 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id= 2023-10-11T11:03:36.273 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 131 = '' 2023-10-11T11:03:36.273 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T11:03:36.273 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.8 2023-10-11T11:03:37.074 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_7 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_7.status 2023-10-11T11:03:37.074 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:03:37.075 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:03:37.075 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_7 2023-10-11T11:03:37.075 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_7.status 2023-10-11T11:03:37.076 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_7 --format xml 2023-10-11T11:03:37.076 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:03:37.133 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T11:03:37.133 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T11:03:37.135 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:-:1.1: Document is empty 2023-10-11T11:03:37.135 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: 2023-10-11T11:03:37.135 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:^ 2023-10-11T11:03:37.136 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id= 2023-10-11T11:03:37.136 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 131 = '' 2023-10-11T11:03:37.136 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T11:03:37.136 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 1.6 2023-10-11T11:03:38.736 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_7 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_7.status 2023-10-11T11:03:38.736 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:03:38.736 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:03:38.737 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_7 2023-10-11T11:03:38.737 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_7.status 2023-10-11T11:03:38.737 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_7 --format xml 2023-10-11T11:03:38.737 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:03:38.789 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T11:03:38.792 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id=131 2023-10-11T11:03:38.792 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 131 = 131 2023-10-11T11:03:38.792 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:03:38.792 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_snap_present cluster1 mirror image_7 snap 2023-10-11T11:03:38.793 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:03:38.793 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:03:38.793 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_7 2023-10-11T11:03:38.793 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap 2023-10-11T11:03:38.793 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T11:03:38.793 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-10-11T11:03:38.794 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 1 2023-10-11T11:03:39.793 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 info mirror/image_7@snap 2023-10-11T11:03:39.840 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:rbd image 'image_7': 2023-10-11T11:03:39.840 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: size 128 MiB in 32 objects 2023-10-11T11:03:39.841 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: order 22 (4 MiB objects) 2023-10-11T11:03:39.841 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: snapshot_count: 2 2023-10-11T11:03:39.841 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: id: 10afc6caf1a4 2023-10-11T11:03:39.841 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: block_name_prefix: rbd_data.10afc6caf1a4 2023-10-11T11:03:39.842 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: format: 2 2023-10-11T11:03:39.842 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-10-11T11:03:39.842 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: op_features: 2023-10-11T11:03:39.842 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: flags: 2023-10-11T11:03:39.843 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: create_timestamp: Wed Oct 11 10:55:04 2023 2023-10-11T11:03:39.843 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: access_timestamp: Wed Oct 11 10:55:04 2023 2023-10-11T11:03:39.843 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: modify_timestamp: Wed Oct 11 10:55:04 2023 2023-10-11T11:03:39.844 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: protected: False 2023-10-11T11:03:39.844 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring state: enabled 2023-10-11T11:03:39.844 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring mode: snapshot 2023-10-11T11:03:39.844 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring global id: 96782a85-6dfc-4ac8-9a0a-1c21292c349e 2023-10-11T11:03:39.845 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring primary: false 2023-10-11T11:03:39.846 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:03:39.846 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ compare_image_snaps mirror image_7 snap 2023-10-11T11:03:39.847 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:03:39.847 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_7 2023-10-11T11:03:39.847 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap 2023-10-11T11:03:39.847 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local ret=0 2023-10-11T11:03:39.848 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local rmt_export=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_7.export 2023-10-11T11:03:39.848 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local loc_export=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_7.export 2023-10-11T11:03:39.848 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rm -f /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_7.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_7.export 2023-10-11T11:03:39.848 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 -p mirror export image_7@snap /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_7.export 2023-10-11T11:03:40.028 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-10-11T11:03:40.034 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 -p mirror export image_7@snap /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_7.export 2023-10-11T11:03:40.267 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-10-11T11:03:40.275 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ cmp /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_7.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_7.export 2023-10-11T11:03:40.329 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rm -f /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_7.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_7.export 2023-10-11T11:03:40.366 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:03:40.367 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T11:03:40.367 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_8 2023-10-11T11:03:40.367 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ create_snap cluster2 mirror image_8 snap 2023-10-11T11:03:40.367 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:03:40.367 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:03:40.368 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_8 2023-10-11T11:03:40.368 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap 2023-10-11T11:03:40.368 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 -p mirror snap create image_8@snap --debug-rbd=20 --debug-journaler=20 2023-10-11T11:03:42.391 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_replay_started cluster1 mirror image_8 2023-10-11T11:03:42.391 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:03:42.392 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:03:42.392 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_8 2023-10-11T11:03:42.392 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_replay_state cluster1 mirror image_8 started 2023-10-11T11:03:42.392 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:03:42.393 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:03:42.393 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_8 2023-10-11T11:03:42.393 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local state=started 2023-10-11T11:03:42.393 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T11:03:42.393 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-10-11T11:03:42.394 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 1 2023-10-11T11:03:43.392 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test_image_replay_state cluster1 mirror image_8 started 2023-10-11T11:03:43.392 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:03:43.392 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:03:43.392 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_8 2023-10-11T11:03:43.393 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local test_state=started 2023-10-11T11:03:43.393 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local status_result 2023-10-11T11:03:43.393 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_state=stopped 2023-10-11T11:03:43.394 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_8 2023-10-11T11:03:43.394 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_instance=cluster1 2023-10-11T11:03:43.394 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ grep -i state 2023-10-11T11:03:43.395 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T11:03:43.395 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:03:43.395 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance=cluster1 2023-10-11T11:03:43.395 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local loop_instance 2023-10-11T11:03:43.395 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-10-11T11:03:43.396 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sleep 0 2023-10-11T11:03:43.396 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-10-11T11:03:43.396 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ seq 0 3 2023-10-11T11:03:43.398 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-10-11T11:03:43.399 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_8 2023-10-11T11:03:43.399 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1:0 2023-10-11T11:03:43.399 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T11:03:43.399 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance 2023-10-11T11:03:43.399 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-10-11T11:03:43.400 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local val=cluster1:0 2023-10-11T11:03:43.400 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_var_name=cluster 2023-10-11T11:03:43.400 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance_var_name=instance 2023-10-11T11:03:43.400 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ cluster=cluster1 2023-10-11T11:03:43.400 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ instance=0 2023-10-11T11:03:43.400 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' 0 = cluster1:0 ']' 2023-10-11T11:03:43.401 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval cluster=cluster1 2023-10-11T11:03:43.401 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:03:43.401 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval instance=0 2023-10-11T11:03:43.401 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=0 2023-10-11T11:03:43.401 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-10-11T11:03:43.402 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local local_cluster=cluster1:0 2023-10-11T11:03:43.402 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster=cluster1 2023-10-11T11:03:43.402 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance 2023-10-11T11:03:43.402 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-10-11T11:03:43.402 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local val=cluster1:0 2023-10-11T11:03:43.402 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster_var_name=local_cluster 2023-10-11T11:03:43.403 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance_var_name=instance 2023-10-11T11:03:43.403 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:03:43.403 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=0 2023-10-11T11:03:43.403 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ '[' 0 = cluster1:0 ']' 2023-10-11T11:03:43.403 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval local_cluster=cluster1 2023-10-11T11:03:43.403 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ local_cluster=cluster1 2023-10-11T11:03:43.404 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval instance=0 2023-10-11T11:03:43.404 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ instance=0 2023-10-11T11:03:43.404 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-10-11T11:03:43.421 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T11:03:43.421 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T11:03:43.422 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T11:03:43.422 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_8 2023-10-11T11:03:43.520 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:Invalid command: unused arguments: ['mirror/image_8'] 2023-10-11T11:03:43.520 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd mirror status : get status for rbd mirror 2023-10-11T11:03:43.520 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:admin_socket: invalid command 2023-10-11T11:03:43.523 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-10-11T11:03:43.523 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_8 2023-10-11T11:03:43.524 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1:1 2023-10-11T11:03:43.524 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T11:03:43.524 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance 2023-10-11T11:03:43.524 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-10-11T11:03:43.524 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local val=cluster1:1 2023-10-11T11:03:43.525 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_var_name=cluster 2023-10-11T11:03:43.525 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance_var_name=instance 2023-10-11T11:03:43.525 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ cluster=cluster1 2023-10-11T11:03:43.525 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ instance=1 2023-10-11T11:03:43.525 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' 1 = cluster1:1 ']' 2023-10-11T11:03:43.525 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval cluster=cluster1 2023-10-11T11:03:43.526 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:03:43.526 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval instance=1 2023-10-11T11:03:43.526 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=1 2023-10-11T11:03:43.526 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-10-11T11:03:43.526 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local local_cluster=cluster1:1 2023-10-11T11:03:43.527 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster=cluster1 2023-10-11T11:03:43.527 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance 2023-10-11T11:03:43.527 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-10-11T11:03:43.527 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local val=cluster1:1 2023-10-11T11:03:43.527 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster_var_name=local_cluster 2023-10-11T11:03:43.528 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance_var_name=instance 2023-10-11T11:03:43.528 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:03:43.528 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=1 2023-10-11T11:03:43.528 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ '[' 1 = cluster1:1 ']' 2023-10-11T11:03:43.528 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval local_cluster=cluster1 2023-10-11T11:03:43.528 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ local_cluster=cluster1 2023-10-11T11:03:43.529 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval instance=1 2023-10-11T11:03:43.529 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ instance=1 2023-10-11T11:03:43.529 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-10-11T11:03:43.542 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T11:03:43.542 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T11:03:43.542 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T11:03:43.543 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_8 2023-10-11T11:03:43.636 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:Invalid command: unused arguments: ['mirror/image_8'] 2023-10-11T11:03:43.636 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd mirror status : get status for rbd mirror 2023-10-11T11:03:43.637 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:admin_socket: invalid command 2023-10-11T11:03:43.640 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-10-11T11:03:43.640 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_8 2023-10-11T11:03:43.640 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1:2 2023-10-11T11:03:43.640 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T11:03:43.640 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance 2023-10-11T11:03:43.641 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-10-11T11:03:43.641 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local val=cluster1:2 2023-10-11T11:03:43.641 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_var_name=cluster 2023-10-11T11:03:43.641 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance_var_name=instance 2023-10-11T11:03:43.641 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ cluster=cluster1 2023-10-11T11:03:43.642 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ instance=2 2023-10-11T11:03:43.642 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' 2 = cluster1:2 ']' 2023-10-11T11:03:43.642 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval cluster=cluster1 2023-10-11T11:03:43.642 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:03:43.643 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval instance=2 2023-10-11T11:03:43.643 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=2 2023-10-11T11:03:43.643 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-10-11T11:03:43.643 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local local_cluster=cluster1:2 2023-10-11T11:03:43.643 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster=cluster1 2023-10-11T11:03:43.644 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance 2023-10-11T11:03:43.644 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-10-11T11:03:43.644 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local val=cluster1:2 2023-10-11T11:03:43.644 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster_var_name=local_cluster 2023-10-11T11:03:43.644 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance_var_name=instance 2023-10-11T11:03:43.644 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:03:43.645 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=2 2023-10-11T11:03:43.645 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ '[' 2 = cluster1:2 ']' 2023-10-11T11:03:43.645 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval local_cluster=cluster1 2023-10-11T11:03:43.645 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ local_cluster=cluster1 2023-10-11T11:03:43.645 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval instance=2 2023-10-11T11:03:43.646 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ instance=2 2023-10-11T11:03:43.646 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-10-11T11:03:43.658 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-10-11T11:03:43.658 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-10-11T11:03:43.658 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-10-11T11:03:43.659 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_8 2023-10-11T11:03:43.808 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ return 0 2023-10-11T11:03:43.809 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ status_result=' "state": "Replaying"' 2023-10-11T11:03:43.810 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ echo ' "state": "Replaying"' 2023-10-11T11:03:43.810 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ grep -i Replaying 2023-10-11T11:03:43.810 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: "state": "Replaying" 2023-10-11T11:03:43.813 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ current_state=started 2023-10-11T11:03:43.813 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test started = started 2023-10-11T11:03:43.813 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:03:43.813 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_8 2023-10-11T11:03:43.814 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local local_cluster=cluster1 2023-10-11T11:03:43.814 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:03:43.814 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:03:43.814 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_8 2023-10-11T11:03:43.814 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' snapshot = journal ']' 2023-10-11T11:03:43.815 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' snapshot = snapshot ']' 2023-10-11T11:03:43.815 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_8 2023-10-11T11:03:43.815 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local local_cluster=cluster1 2023-10-11T11:03:43.815 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:03:43.816 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:03:43.816 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_8 2023-10-11T11:03:43.816 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ mkfname cluster2-mirror-image_8.status 2023-10-11T11:03:43.816 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ echo cluster2-mirror-image_8.status 2023-10-11T11:03:43.816 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -e 's|[/ ]|_|g' 2023-10-11T11:03:43.817 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local status_log=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_8.status 2023-10-11T11:03:43.817 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ mkfname cluster1-mirror-image_8.status 2023-10-11T11:03:43.817 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ echo cluster1-mirror-image_8.status 2023-10-11T11:03:43.817 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -e 's|[/ ]|_|g' 2023-10-11T11:03:43.818 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local local_status_log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_8.status 2023-10-11T11:03:43.819 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ mirror_image_snapshot cluster2 mirror image_8 2023-10-11T11:03:43.819 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:03:43.819 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:03:43.819 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_8 2023-10-11T11:03:43.819 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_8 2023-10-11T11:03:44.392 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:Snapshot ID: 133 2023-10-11T11:03:44.403 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_8 /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_8.status 2023-10-11T11:03:44.403 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:03:44.403 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:03:44.403 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_8 2023-10-11T11:03:44.404 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_8.status 2023-10-11T11:03:44.404 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_8 --format xml 2023-10-11T11:03:44.404 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:03:44.447 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-10-11T11:03:44.448 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snapshot_id=133 2023-10-11T11:03:44.448 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T11:03:44.449 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T11:03:44.449 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.2 2023-10-11T11:03:44.650 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_8 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_8.status 2023-10-11T11:03:44.650 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:03:44.651 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:03:44.651 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_8 2023-10-11T11:03:44.651 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_8.status 2023-10-11T11:03:44.651 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_8 --format xml 2023-10-11T11:03:44.652 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:03:44.697 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T11:03:44.697 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T11:03:44.699 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:-:1.1: Document is empty 2023-10-11T11:03:44.699 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: 2023-10-11T11:03:44.699 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:^ 2023-10-11T11:03:44.700 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id= 2023-10-11T11:03:44.700 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 133 = '' 2023-10-11T11:03:44.700 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T11:03:44.700 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.4 2023-10-11T11:03:45.100 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_8 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_8.status 2023-10-11T11:03:45.101 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:03:45.101 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:03:45.101 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_8 2023-10-11T11:03:45.101 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_8.status 2023-10-11T11:03:45.102 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_8 --format xml 2023-10-11T11:03:45.102 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:03:45.147 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T11:03:45.147 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T11:03:45.149 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:-:1.1: Document is empty 2023-10-11T11:03:45.149 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: 2023-10-11T11:03:45.150 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:^ 2023-10-11T11:03:45.150 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id= 2023-10-11T11:03:45.150 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 133 = '' 2023-10-11T11:03:45.150 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T11:03:45.151 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.8 2023-10-11T11:03:45.950 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_8 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_8.status 2023-10-11T11:03:45.950 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:03:45.951 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:03:45.951 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_8 2023-10-11T11:03:45.951 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_8.status 2023-10-11T11:03:45.951 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_8 --format xml 2023-10-11T11:03:45.951 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:03:45.996 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T11:03:45.998 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id=133 2023-10-11T11:03:45.998 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 133 = 133 2023-10-11T11:03:45.999 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:03:45.999 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_snap_present cluster1 mirror image_8 snap 2023-10-11T11:03:45.999 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:03:45.999 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:03:45.999 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_8 2023-10-11T11:03:46.000 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap 2023-10-11T11:03:46.000 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T11:03:46.000 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-10-11T11:03:46.000 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 1 2023-10-11T11:03:46.999 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 info mirror/image_8@snap 2023-10-11T11:03:47.049 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:rbd image 'image_8': 2023-10-11T11:03:47.049 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: size 128 MiB in 32 objects 2023-10-11T11:03:47.049 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: order 22 (4 MiB objects) 2023-10-11T11:03:47.049 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: snapshot_count: 2 2023-10-11T11:03:47.050 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: id: 123d9f0970ca 2023-10-11T11:03:47.050 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: block_name_prefix: rbd_data.123d9f0970ca 2023-10-11T11:03:47.050 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: format: 2 2023-10-11T11:03:47.050 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-10-11T11:03:47.050 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: op_features: 2023-10-11T11:03:47.050 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: flags: 2023-10-11T11:03:47.051 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: create_timestamp: Wed Oct 11 10:55:16 2023 2023-10-11T11:03:47.051 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: access_timestamp: Wed Oct 11 10:55:16 2023 2023-10-11T11:03:47.051 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: modify_timestamp: Wed Oct 11 10:55:16 2023 2023-10-11T11:03:47.051 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: protected: False 2023-10-11T11:03:47.051 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring state: enabled 2023-10-11T11:03:47.051 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring mode: snapshot 2023-10-11T11:03:47.052 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring global id: ff8c48f9-2b0b-4783-bc85-bfb3318837fb 2023-10-11T11:03:47.052 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring primary: false 2023-10-11T11:03:47.053 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:03:47.053 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ compare_image_snaps mirror image_8 snap 2023-10-11T11:03:47.053 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:03:47.053 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_8 2023-10-11T11:03:47.053 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap 2023-10-11T11:03:47.054 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local ret=0 2023-10-11T11:03:47.054 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local rmt_export=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_8.export 2023-10-11T11:03:47.054 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local loc_export=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_8.export 2023-10-11T11:03:47.054 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rm -f /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_8.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_8.export 2023-10-11T11:03:47.055 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 -p mirror export image_8@snap /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_8.export 2023-10-11T11:03:47.238 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-10-11T11:03:47.245 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 -p mirror export image_8@snap /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_8.export 2023-10-11T11:03:47.481 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-10-11T11:03:47.487 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ cmp /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_8.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_8.export 2023-10-11T11:03:47.543 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rm -f /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_8.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_8.export 2023-10-11T11:03:47.580 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:03:47.580 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T11:03:47.580 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_9 2023-10-11T11:03:47.580 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ create_snap cluster2 mirror image_9 snap 2023-10-11T11:03:47.581 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:03:47.581 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:03:47.581 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_9 2023-10-11T11:03:47.581 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap 2023-10-11T11:03:47.581 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 -p mirror snap create image_9@snap --debug-rbd=20 --debug-journaler=20 2023-10-11T11:03:49.474 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_replay_started cluster1 mirror image_9 2023-10-11T11:03:49.474 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:03:49.475 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:03:49.475 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_9 2023-10-11T11:03:49.475 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_replay_state cluster1 mirror image_9 started 2023-10-11T11:03:49.475 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:03:49.475 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:03:49.476 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_9 2023-10-11T11:03:49.476 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local state=started 2023-10-11T11:03:49.476 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T11:03:49.476 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-10-11T11:03:49.476 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 1 2023-10-11T11:03:50.476 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test_image_replay_state cluster1 mirror image_9 started 2023-10-11T11:03:50.477 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:03:50.477 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:03:50.477 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_9 2023-10-11T11:03:50.477 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local test_state=started 2023-10-11T11:03:50.477 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local status_result 2023-10-11T11:03:50.478 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_state=stopped 2023-10-11T11:03:50.478 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_9 2023-10-11T11:03:50.478 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_instance=cluster1 2023-10-11T11:03:50.478 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T11:03:50.478 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ grep -i state 2023-10-11T11:03:50.479 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:03:50.479 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance=cluster1 2023-10-11T11:03:50.479 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local loop_instance 2023-10-11T11:03:50.479 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-10-11T11:03:50.479 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sleep 0 2023-10-11T11:03:50.481 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-10-11T11:03:50.482 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ seq 0 3 2023-10-11T11:03:50.482 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-10-11T11:03:50.482 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_9 2023-10-11T11:03:50.482 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1:0 2023-10-11T11:03:50.483 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T11:03:50.483 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance 2023-10-11T11:03:50.483 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-10-11T11:03:50.483 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local val=cluster1:0 2023-10-11T11:03:50.483 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_var_name=cluster 2023-10-11T11:03:50.483 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance_var_name=instance 2023-10-11T11:03:50.484 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ cluster=cluster1 2023-10-11T11:03:50.484 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ instance=0 2023-10-11T11:03:50.484 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' 0 = cluster1:0 ']' 2023-10-11T11:03:50.484 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval cluster=cluster1 2023-10-11T11:03:50.484 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:03:50.485 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval instance=0 2023-10-11T11:03:50.485 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=0 2023-10-11T11:03:50.485 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-10-11T11:03:50.485 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local local_cluster=cluster1:0 2023-10-11T11:03:50.485 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster=cluster1 2023-10-11T11:03:50.486 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance 2023-10-11T11:03:50.486 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-10-11T11:03:50.486 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local val=cluster1:0 2023-10-11T11:03:50.486 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster_var_name=local_cluster 2023-10-11T11:03:50.487 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance_var_name=instance 2023-10-11T11:03:50.487 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:03:50.488 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=0 2023-10-11T11:03:50.488 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ '[' 0 = cluster1:0 ']' 2023-10-11T11:03:50.488 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval local_cluster=cluster1 2023-10-11T11:03:50.488 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ local_cluster=cluster1 2023-10-11T11:03:50.488 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval instance=0 2023-10-11T11:03:50.489 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ instance=0 2023-10-11T11:03:50.489 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-10-11T11:03:50.503 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T11:03:50.503 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T11:03:50.503 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T11:03:50.503 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_9 2023-10-11T11:03:50.610 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:Invalid command: unused arguments: ['mirror/image_9'] 2023-10-11T11:03:50.610 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd mirror status : get status for rbd mirror 2023-10-11T11:03:50.611 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:admin_socket: invalid command 2023-10-11T11:03:50.613 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-10-11T11:03:50.613 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_9 2023-10-11T11:03:50.614 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1:1 2023-10-11T11:03:50.614 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T11:03:50.614 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance 2023-10-11T11:03:50.614 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-10-11T11:03:50.614 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local val=cluster1:1 2023-10-11T11:03:50.615 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_var_name=cluster 2023-10-11T11:03:50.615 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance_var_name=instance 2023-10-11T11:03:50.615 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ cluster=cluster1 2023-10-11T11:03:50.615 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ instance=1 2023-10-11T11:03:50.615 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' 1 = cluster1:1 ']' 2023-10-11T11:03:50.616 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval cluster=cluster1 2023-10-11T11:03:50.616 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:03:50.616 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval instance=1 2023-10-11T11:03:50.616 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=1 2023-10-11T11:03:50.616 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-10-11T11:03:50.617 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local local_cluster=cluster1:1 2023-10-11T11:03:50.617 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster=cluster1 2023-10-11T11:03:50.617 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance 2023-10-11T11:03:50.617 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-10-11T11:03:50.617 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local val=cluster1:1 2023-10-11T11:03:50.617 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster_var_name=local_cluster 2023-10-11T11:03:50.618 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance_var_name=instance 2023-10-11T11:03:50.618 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:03:50.618 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=1 2023-10-11T11:03:50.618 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ '[' 1 = cluster1:1 ']' 2023-10-11T11:03:50.619 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval local_cluster=cluster1 2023-10-11T11:03:50.619 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ local_cluster=cluster1 2023-10-11T11:03:50.619 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval instance=1 2023-10-11T11:03:50.619 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ instance=1 2023-10-11T11:03:50.619 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-10-11T11:03:50.633 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T11:03:50.633 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T11:03:50.634 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T11:03:50.634 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_9 2023-10-11T11:03:50.730 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:Invalid command: unused arguments: ['mirror/image_9'] 2023-10-11T11:03:50.730 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd mirror status : get status for rbd mirror 2023-10-11T11:03:50.730 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:admin_socket: invalid command 2023-10-11T11:03:50.733 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-10-11T11:03:50.734 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_9 2023-10-11T11:03:50.734 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1:2 2023-10-11T11:03:50.734 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T11:03:50.734 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance 2023-10-11T11:03:50.734 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-10-11T11:03:50.735 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local val=cluster1:2 2023-10-11T11:03:50.735 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_var_name=cluster 2023-10-11T11:03:50.735 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance_var_name=instance 2023-10-11T11:03:50.735 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ cluster=cluster1 2023-10-11T11:03:50.735 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ instance=2 2023-10-11T11:03:50.736 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' 2 = cluster1:2 ']' 2023-10-11T11:03:50.736 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval cluster=cluster1 2023-10-11T11:03:50.736 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:03:50.736 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval instance=2 2023-10-11T11:03:50.736 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=2 2023-10-11T11:03:50.737 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-10-11T11:03:50.737 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local local_cluster=cluster1:2 2023-10-11T11:03:50.737 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster=cluster1 2023-10-11T11:03:50.737 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance 2023-10-11T11:03:50.737 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-10-11T11:03:50.738 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local val=cluster1:2 2023-10-11T11:03:50.738 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster_var_name=local_cluster 2023-10-11T11:03:50.738 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance_var_name=instance 2023-10-11T11:03:50.738 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:03:50.738 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=2 2023-10-11T11:03:50.739 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ '[' 2 = cluster1:2 ']' 2023-10-11T11:03:50.739 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval local_cluster=cluster1 2023-10-11T11:03:50.739 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ local_cluster=cluster1 2023-10-11T11:03:50.739 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval instance=2 2023-10-11T11:03:50.739 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ instance=2 2023-10-11T11:03:50.740 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-10-11T11:03:50.752 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-10-11T11:03:50.752 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-10-11T11:03:50.752 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-10-11T11:03:50.753 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_9 2023-10-11T11:03:50.906 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ return 0 2023-10-11T11:03:50.907 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ status_result=' "state": "Replaying"' 2023-10-11T11:03:50.907 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ echo ' "state": "Replaying"' 2023-10-11T11:03:50.907 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ grep -i Replaying 2023-10-11T11:03:50.908 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: "state": "Replaying" 2023-10-11T11:03:50.909 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ current_state=started 2023-10-11T11:03:50.909 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test started = started 2023-10-11T11:03:50.910 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:03:50.910 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_9 2023-10-11T11:03:50.910 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local local_cluster=cluster1 2023-10-11T11:03:50.910 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:03:50.910 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:03:50.910 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_9 2023-10-11T11:03:50.911 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' snapshot = journal ']' 2023-10-11T11:03:50.911 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' snapshot = snapshot ']' 2023-10-11T11:03:50.911 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_9 2023-10-11T11:03:50.911 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local local_cluster=cluster1 2023-10-11T11:03:50.911 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:03:50.912 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:03:50.912 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_9 2023-10-11T11:03:50.912 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ mkfname cluster2-mirror-image_9.status 2023-10-11T11:03:50.912 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ echo cluster2-mirror-image_9.status 2023-10-11T11:03:50.912 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -e 's|[/ ]|_|g' 2023-10-11T11:03:50.913 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local status_log=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_9.status 2023-10-11T11:03:50.913 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ mkfname cluster1-mirror-image_9.status 2023-10-11T11:03:50.914 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ echo cluster1-mirror-image_9.status 2023-10-11T11:03:50.914 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -e 's|[/ ]|_|g' 2023-10-11T11:03:50.914 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local local_status_log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_9.status 2023-10-11T11:03:50.914 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ mirror_image_snapshot cluster2 mirror image_9 2023-10-11T11:03:50.914 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:03:50.915 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:03:50.915 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_9 2023-10-11T11:03:50.915 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_9 2023-10-11T11:03:51.472 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:Snapshot ID: 135 2023-10-11T11:03:51.483 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_9 /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_9.status 2023-10-11T11:03:51.483 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:03:51.484 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:03:51.484 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_9 2023-10-11T11:03:51.484 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_9.status 2023-10-11T11:03:51.484 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_9 --format xml 2023-10-11T11:03:51.485 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:03:51.525 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-10-11T11:03:51.527 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snapshot_id=135 2023-10-11T11:03:51.527 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T11:03:51.527 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T11:03:51.528 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.2 2023-10-11T11:03:51.728 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_9 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_9.status 2023-10-11T11:03:51.728 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:03:51.728 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:03:51.728 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_9 2023-10-11T11:03:51.729 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_9.status 2023-10-11T11:03:51.729 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_9 --format xml 2023-10-11T11:03:51.729 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:03:51.773 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T11:03:51.773 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T11:03:51.775 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:-:1.1: Document is empty 2023-10-11T11:03:51.775 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: 2023-10-11T11:03:51.775 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:^ 2023-10-11T11:03:51.775 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id= 2023-10-11T11:03:51.776 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 135 = '' 2023-10-11T11:03:51.776 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T11:03:51.776 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.4 2023-10-11T11:03:52.177 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_9 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_9.status 2023-10-11T11:03:52.177 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:03:52.177 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:03:52.177 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_9 2023-10-11T11:03:52.177 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_9.status 2023-10-11T11:03:52.178 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_9 --format xml 2023-10-11T11:03:52.178 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:03:52.222 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T11:03:52.222 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T11:03:52.224 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:-:1.1: Document is empty 2023-10-11T11:03:52.224 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: 2023-10-11T11:03:52.224 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:^ 2023-10-11T11:03:52.224 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id= 2023-10-11T11:03:52.224 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 135 = '' 2023-10-11T11:03:52.225 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T11:03:52.225 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.8 2023-10-11T11:03:53.025 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_9 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_9.status 2023-10-11T11:03:53.026 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:03:53.026 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:03:53.026 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_9 2023-10-11T11:03:53.026 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_9.status 2023-10-11T11:03:53.026 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_9 --format xml 2023-10-11T11:03:53.027 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:03:53.069 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T11:03:53.072 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id=135 2023-10-11T11:03:53.072 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 135 = 135 2023-10-11T11:03:53.072 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:03:53.072 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_snap_present cluster1 mirror image_9 snap 2023-10-11T11:03:53.073 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:03:53.073 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:03:53.073 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_9 2023-10-11T11:03:53.073 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap 2023-10-11T11:03:53.073 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T11:03:53.073 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-10-11T11:03:53.074 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 1 2023-10-11T11:03:54.072 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 info mirror/image_9@snap 2023-10-11T11:03:54.118 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:rbd image 'image_9': 2023-10-11T11:03:54.118 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: size 128 MiB in 32 objects 2023-10-11T11:03:54.118 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: order 22 (4 MiB objects) 2023-10-11T11:03:54.119 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: snapshot_count: 2 2023-10-11T11:03:54.119 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: id: 123df613ad30 2023-10-11T11:03:54.119 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: block_name_prefix: rbd_data.123df613ad30 2023-10-11T11:03:54.119 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: format: 2 2023-10-11T11:03:54.120 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-10-11T11:03:54.120 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: op_features: 2023-10-11T11:03:54.120 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: flags: 2023-10-11T11:03:54.120 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: create_timestamp: Wed Oct 11 10:55:28 2023 2023-10-11T11:03:54.121 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: access_timestamp: Wed Oct 11 10:55:28 2023 2023-10-11T11:03:54.121 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: modify_timestamp: Wed Oct 11 10:55:28 2023 2023-10-11T11:03:54.121 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: protected: False 2023-10-11T11:03:54.121 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring state: enabled 2023-10-11T11:03:54.122 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring mode: snapshot 2023-10-11T11:03:54.122 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring global id: ca5ef4e0-7cfe-484d-8d61-93b7e745a919 2023-10-11T11:03:54.122 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring primary: false 2023-10-11T11:03:54.124 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:03:54.124 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ compare_image_snaps mirror image_9 snap 2023-10-11T11:03:54.124 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:03:54.124 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_9 2023-10-11T11:03:54.125 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap 2023-10-11T11:03:54.125 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local ret=0 2023-10-11T11:03:54.125 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local rmt_export=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_9.export 2023-10-11T11:03:54.126 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local loc_export=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_9.export 2023-10-11T11:03:54.126 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rm -f /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_9.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_9.export 2023-10-11T11:03:54.126 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 -p mirror export image_9@snap /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_9.export 2023-10-11T11:03:54.301 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-10-11T11:03:54.308 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 -p mirror export image_9@snap /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_9.export 2023-10-11T11:03:54.554 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-10-11T11:03:54.560 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ cmp /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_9.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_9.export 2023-10-11T11:03:54.614 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rm -f /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_9.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_9.export 2023-10-11T11:03:54.652 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:03:54.652 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T11:03:54.652 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_10 2023-10-11T11:03:54.653 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ create_snap cluster2 mirror image_10 snap 2023-10-11T11:03:54.653 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:03:54.653 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:03:54.653 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_10 2023-10-11T11:03:54.654 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap 2023-10-11T11:03:54.654 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 -p mirror snap create image_10@snap --debug-rbd=20 --debug-journaler=20 2023-10-11T11:03:56.496 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_replay_started cluster1 mirror image_10 2023-10-11T11:03:56.496 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:03:56.497 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:03:56.497 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_10 2023-10-11T11:03:56.497 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_replay_state cluster1 mirror image_10 started 2023-10-11T11:03:56.497 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:03:56.497 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:03:56.498 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_10 2023-10-11T11:03:56.498 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local state=started 2023-10-11T11:03:56.498 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T11:03:56.498 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-10-11T11:03:56.498 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 1 2023-10-11T11:03:57.497 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test_image_replay_state cluster1 mirror image_10 started 2023-10-11T11:03:57.498 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:03:57.498 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:03:57.498 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_10 2023-10-11T11:03:57.498 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local test_state=started 2023-10-11T11:03:57.498 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local status_result 2023-10-11T11:03:57.498 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_state=stopped 2023-10-11T11:03:57.499 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_10 2023-10-11T11:03:57.500 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_instance=cluster1 2023-10-11T11:03:57.500 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T11:03:57.500 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ grep -i state 2023-10-11T11:03:57.500 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:03:57.500 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance=cluster1 2023-10-11T11:03:57.501 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local loop_instance 2023-10-11T11:03:57.501 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-10-11T11:03:57.501 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sleep 0 2023-10-11T11:03:57.501 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-10-11T11:03:57.501 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ seq 0 3 2023-10-11T11:03:57.503 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-10-11T11:03:57.504 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_10 2023-10-11T11:03:57.504 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1:0 2023-10-11T11:03:57.504 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T11:03:57.504 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance 2023-10-11T11:03:57.504 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-10-11T11:03:57.505 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local val=cluster1:0 2023-10-11T11:03:57.505 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_var_name=cluster 2023-10-11T11:03:57.505 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance_var_name=instance 2023-10-11T11:03:57.505 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ cluster=cluster1 2023-10-11T11:03:57.505 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ instance=0 2023-10-11T11:03:57.505 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' 0 = cluster1:0 ']' 2023-10-11T11:03:57.506 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval cluster=cluster1 2023-10-11T11:03:57.506 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:03:57.506 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval instance=0 2023-10-11T11:03:57.506 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=0 2023-10-11T11:03:57.506 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-10-11T11:03:57.507 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local local_cluster=cluster1:0 2023-10-11T11:03:57.507 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster=cluster1 2023-10-11T11:03:57.507 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance 2023-10-11T11:03:57.507 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-10-11T11:03:57.508 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local val=cluster1:0 2023-10-11T11:03:57.508 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster_var_name=local_cluster 2023-10-11T11:03:57.509 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance_var_name=instance 2023-10-11T11:03:57.509 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:03:57.509 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=0 2023-10-11T11:03:57.509 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ '[' 0 = cluster1:0 ']' 2023-10-11T11:03:57.509 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval local_cluster=cluster1 2023-10-11T11:03:57.509 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ local_cluster=cluster1 2023-10-11T11:03:57.510 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval instance=0 2023-10-11T11:03:57.510 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ instance=0 2023-10-11T11:03:57.510 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-10-11T11:03:57.526 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T11:03:57.526 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T11:03:57.526 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T11:03:57.527 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_10 2023-10-11T11:03:57.626 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:Invalid command: unused arguments: ['mirror/image_10'] 2023-10-11T11:03:57.627 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd mirror status : get status for rbd mirror 2023-10-11T11:03:57.627 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:admin_socket: invalid command 2023-10-11T11:03:57.630 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-10-11T11:03:57.630 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_10 2023-10-11T11:03:57.630 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1:1 2023-10-11T11:03:57.630 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T11:03:57.631 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance 2023-10-11T11:03:57.631 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-10-11T11:03:57.631 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local val=cluster1:1 2023-10-11T11:03:57.631 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_var_name=cluster 2023-10-11T11:03:57.631 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance_var_name=instance 2023-10-11T11:03:57.632 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ cluster=cluster1 2023-10-11T11:03:57.632 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ instance=1 2023-10-11T11:03:57.632 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' 1 = cluster1:1 ']' 2023-10-11T11:03:57.632 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval cluster=cluster1 2023-10-11T11:03:57.632 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:03:57.632 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval instance=1 2023-10-11T11:03:57.633 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=1 2023-10-11T11:03:57.633 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-10-11T11:03:57.633 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local local_cluster=cluster1:1 2023-10-11T11:03:57.633 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster=cluster1 2023-10-11T11:03:57.633 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance 2023-10-11T11:03:57.633 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-10-11T11:03:57.634 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local val=cluster1:1 2023-10-11T11:03:57.634 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster_var_name=local_cluster 2023-10-11T11:03:57.634 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance_var_name=instance 2023-10-11T11:03:57.634 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:03:57.634 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=1 2023-10-11T11:03:57.635 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ '[' 1 = cluster1:1 ']' 2023-10-11T11:03:57.635 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval local_cluster=cluster1 2023-10-11T11:03:57.635 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ local_cluster=cluster1 2023-10-11T11:03:57.635 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval instance=1 2023-10-11T11:03:57.635 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ instance=1 2023-10-11T11:03:57.636 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-10-11T11:03:57.648 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T11:03:57.649 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T11:03:57.649 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T11:03:57.649 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_10 2023-10-11T11:03:57.743 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:Invalid command: unused arguments: ['mirror/image_10'] 2023-10-11T11:03:57.744 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd mirror status : get status for rbd mirror 2023-10-11T11:03:57.744 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:admin_socket: invalid command 2023-10-11T11:03:57.747 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-10-11T11:03:57.747 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_10 2023-10-11T11:03:57.747 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1:2 2023-10-11T11:03:57.747 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T11:03:57.747 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance 2023-10-11T11:03:57.748 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-10-11T11:03:57.748 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local val=cluster1:2 2023-10-11T11:03:57.748 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_var_name=cluster 2023-10-11T11:03:57.748 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance_var_name=instance 2023-10-11T11:03:57.748 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ cluster=cluster1 2023-10-11T11:03:57.748 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ instance=2 2023-10-11T11:03:57.749 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' 2 = cluster1:2 ']' 2023-10-11T11:03:57.749 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval cluster=cluster1 2023-10-11T11:03:57.749 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:03:57.749 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval instance=2 2023-10-11T11:03:57.749 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=2 2023-10-11T11:03:57.749 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-10-11T11:03:57.750 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local local_cluster=cluster1:2 2023-10-11T11:03:57.750 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster=cluster1 2023-10-11T11:03:57.750 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance 2023-10-11T11:03:57.750 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-10-11T11:03:57.750 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local val=cluster1:2 2023-10-11T11:03:57.751 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster_var_name=local_cluster 2023-10-11T11:03:57.751 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance_var_name=instance 2023-10-11T11:03:57.751 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:03:57.751 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=2 2023-10-11T11:03:57.751 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ '[' 2 = cluster1:2 ']' 2023-10-11T11:03:57.751 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval local_cluster=cluster1 2023-10-11T11:03:57.752 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ local_cluster=cluster1 2023-10-11T11:03:57.752 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval instance=2 2023-10-11T11:03:57.752 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ instance=2 2023-10-11T11:03:57.752 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-10-11T11:03:57.766 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-10-11T11:03:57.766 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-10-11T11:03:57.766 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-10-11T11:03:57.766 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_10 2023-10-11T11:03:57.914 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ return 0 2023-10-11T11:03:57.915 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ status_result=' "state": "Replaying"' 2023-10-11T11:03:57.915 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ echo ' "state": "Replaying"' 2023-10-11T11:03:57.916 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ grep -i Replaying 2023-10-11T11:03:57.916 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: "state": "Replaying" 2023-10-11T11:03:57.917 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ current_state=started 2023-10-11T11:03:57.918 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test started = started 2023-10-11T11:03:57.918 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:03:57.918 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_10 2023-10-11T11:03:57.918 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local local_cluster=cluster1 2023-10-11T11:03:57.918 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:03:57.919 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:03:57.919 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_10 2023-10-11T11:03:57.919 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' snapshot = journal ']' 2023-10-11T11:03:57.919 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' snapshot = snapshot ']' 2023-10-11T11:03:57.919 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_10 2023-10-11T11:03:57.919 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local local_cluster=cluster1 2023-10-11T11:03:57.920 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:03:57.920 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:03:57.920 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_10 2023-10-11T11:03:57.920 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ mkfname cluster2-mirror-image_10.status 2023-10-11T11:03:57.920 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ echo cluster2-mirror-image_10.status 2023-10-11T11:03:57.920 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -e 's|[/ ]|_|g' 2023-10-11T11:03:57.921 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local status_log=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_10.status 2023-10-11T11:03:57.922 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ mkfname cluster1-mirror-image_10.status 2023-10-11T11:03:57.922 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ echo cluster1-mirror-image_10.status 2023-10-11T11:03:57.922 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -e 's|[/ ]|_|g' 2023-10-11T11:03:57.922 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local local_status_log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_10.status 2023-10-11T11:03:57.922 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ mirror_image_snapshot cluster2 mirror image_10 2023-10-11T11:03:57.923 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:03:57.923 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:03:57.923 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_10 2023-10-11T11:03:57.923 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_10 2023-10-11T11:03:58.495 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:Snapshot ID: 137 2023-10-11T11:03:58.507 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_10 /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_10.status 2023-10-11T11:03:58.507 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:03:58.507 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:03:58.507 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_10 2023-10-11T11:03:58.508 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_10.status 2023-10-11T11:03:58.508 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_10 --format xml 2023-10-11T11:03:58.508 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:03:58.550 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-10-11T11:03:58.553 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snapshot_id=137 2023-10-11T11:03:58.553 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T11:03:58.553 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T11:03:58.553 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.2 2023-10-11T11:03:58.754 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_10 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_10.status 2023-10-11T11:03:58.754 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:03:58.754 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:03:58.754 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_10 2023-10-11T11:03:58.755 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_10.status 2023-10-11T11:03:58.755 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_10 --format xml 2023-10-11T11:03:58.755 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:03:58.800 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T11:03:58.801 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T11:03:58.802 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:-:1.1: Document is empty 2023-10-11T11:03:58.803 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: 2023-10-11T11:03:58.803 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:^ 2023-10-11T11:03:58.803 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id= 2023-10-11T11:03:58.803 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 137 = '' 2023-10-11T11:03:58.803 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T11:03:58.803 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.4 2023-10-11T11:03:59.204 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_10 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_10.status 2023-10-11T11:03:59.204 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:03:59.204 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:03:59.204 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_10 2023-10-11T11:03:59.205 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_10.status 2023-10-11T11:03:59.205 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_10 --format xml 2023-10-11T11:03:59.205 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:03:59.250 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T11:03:59.250 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T11:03:59.252 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:-:1.1: Document is empty 2023-10-11T11:03:59.252 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: 2023-10-11T11:03:59.252 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:^ 2023-10-11T11:03:59.252 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id= 2023-10-11T11:03:59.253 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 137 = '' 2023-10-11T11:03:59.253 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T11:03:59.253 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.8 2023-10-11T11:04:00.054 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_10 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_10.status 2023-10-11T11:04:00.054 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:04:00.054 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:04:00.054 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_10 2023-10-11T11:04:00.055 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_10.status 2023-10-11T11:04:00.055 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_10 --format xml 2023-10-11T11:04:00.055 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:04:00.098 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T11:04:00.100 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id=137 2023-10-11T11:04:00.100 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 137 = 137 2023-10-11T11:04:00.101 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:04:00.101 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_snap_present cluster1 mirror image_10 snap 2023-10-11T11:04:00.101 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:04:00.101 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:04:00.101 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_10 2023-10-11T11:04:00.101 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap 2023-10-11T11:04:00.102 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T11:04:00.102 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-10-11T11:04:00.102 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 1 2023-10-11T11:04:01.101 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 info mirror/image_10@snap 2023-10-11T11:04:01.148 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:rbd image 'image_10': 2023-10-11T11:04:01.148 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: size 128 MiB in 32 objects 2023-10-11T11:04:01.148 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: order 22 (4 MiB objects) 2023-10-11T11:04:01.148 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: snapshot_count: 2 2023-10-11T11:04:01.149 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: id: 123de041f11 2023-10-11T11:04:01.149 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: block_name_prefix: rbd_data.123de041f11 2023-10-11T11:04:01.149 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: format: 2 2023-10-11T11:04:01.149 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-10-11T11:04:01.149 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: op_features: 2023-10-11T11:04:01.150 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: flags: 2023-10-11T11:04:01.150 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: create_timestamp: Wed Oct 11 10:55:34 2023 2023-10-11T11:04:01.150 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: access_timestamp: Wed Oct 11 10:55:34 2023 2023-10-11T11:04:01.150 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: modify_timestamp: Wed Oct 11 10:55:34 2023 2023-10-11T11:04:01.150 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: protected: False 2023-10-11T11:04:01.150 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring state: enabled 2023-10-11T11:04:01.151 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring mode: snapshot 2023-10-11T11:04:01.151 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring global id: 965cf2d0-179c-4721-be88-950f8a10b7ae 2023-10-11T11:04:01.151 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring primary: false 2023-10-11T11:04:01.152 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:04:01.152 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ compare_image_snaps mirror image_10 snap 2023-10-11T11:04:01.152 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:04:01.152 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_10 2023-10-11T11:04:01.153 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap 2023-10-11T11:04:01.153 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local ret=0 2023-10-11T11:04:01.153 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local rmt_export=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_10.export 2023-10-11T11:04:01.153 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local loc_export=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_10.export 2023-10-11T11:04:01.153 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rm -f /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_10.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_10.export 2023-10-11T11:04:01.154 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 -p mirror export image_10@snap /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_10.export 2023-10-11T11:04:01.337 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-10-11T11:04:01.343 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 -p mirror export image_10@snap /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_10.export 2023-10-11T11:04:01.559 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-10-11T11:04:01.566 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ cmp /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_10.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_10.export 2023-10-11T11:04:01.621 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rm -f /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_10.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_10.export 2023-10-11T11:04:01.658 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:04:01.658 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T11:04:01.658 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_11 2023-10-11T11:04:01.659 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ create_snap cluster2 mirror image_11 snap 2023-10-11T11:04:01.659 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:04:01.659 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:04:01.659 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_11 2023-10-11T11:04:01.659 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap 2023-10-11T11:04:01.659 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 -p mirror snap create image_11@snap --debug-rbd=20 --debug-journaler=20 2023-10-11T11:04:03.512 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_replay_started cluster1 mirror image_11 2023-10-11T11:04:03.512 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:04:03.513 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:04:03.513 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_11 2023-10-11T11:04:03.513 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_replay_state cluster1 mirror image_11 started 2023-10-11T11:04:03.513 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:04:03.514 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:04:03.514 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_11 2023-10-11T11:04:03.514 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local state=started 2023-10-11T11:04:03.514 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T11:04:03.515 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-10-11T11:04:03.515 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 1 2023-10-11T11:04:04.514 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test_image_replay_state cluster1 mirror image_11 started 2023-10-11T11:04:04.515 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:04:04.515 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:04:04.515 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_11 2023-10-11T11:04:04.515 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local test_state=started 2023-10-11T11:04:04.516 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local status_result 2023-10-11T11:04:04.516 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_state=stopped 2023-10-11T11:04:04.516 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_11 2023-10-11T11:04:04.517 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_instance=cluster1 2023-10-11T11:04:04.517 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T11:04:04.517 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:04:04.517 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance=cluster1 2023-10-11T11:04:04.518 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ grep -i state 2023-10-11T11:04:04.518 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local loop_instance 2023-10-11T11:04:04.518 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-10-11T11:04:04.518 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sleep 0 2023-10-11T11:04:04.522 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-10-11T11:04:04.523 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ seq 0 3 2023-10-11T11:04:04.523 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-10-11T11:04:04.523 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_11 2023-10-11T11:04:04.523 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1:0 2023-10-11T11:04:04.524 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T11:04:04.524 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance 2023-10-11T11:04:04.524 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-10-11T11:04:04.524 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local val=cluster1:0 2023-10-11T11:04:04.525 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_var_name=cluster 2023-10-11T11:04:04.525 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance_var_name=instance 2023-10-11T11:04:04.525 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ cluster=cluster1 2023-10-11T11:04:04.526 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ instance=0 2023-10-11T11:04:04.526 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' 0 = cluster1:0 ']' 2023-10-11T11:04:04.526 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval cluster=cluster1 2023-10-11T11:04:04.526 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:04:04.527 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval instance=0 2023-10-11T11:04:04.527 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=0 2023-10-11T11:04:04.527 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-10-11T11:04:04.527 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local local_cluster=cluster1:0 2023-10-11T11:04:04.528 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster=cluster1 2023-10-11T11:04:04.528 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance 2023-10-11T11:04:04.528 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-10-11T11:04:04.528 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local val=cluster1:0 2023-10-11T11:04:04.528 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster_var_name=local_cluster 2023-10-11T11:04:04.529 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance_var_name=instance 2023-10-11T11:04:04.529 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:04:04.529 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=0 2023-10-11T11:04:04.529 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ '[' 0 = cluster1:0 ']' 2023-10-11T11:04:04.530 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval local_cluster=cluster1 2023-10-11T11:04:04.530 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ local_cluster=cluster1 2023-10-11T11:04:04.530 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval instance=0 2023-10-11T11:04:04.530 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ instance=0 2023-10-11T11:04:04.531 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-10-11T11:04:04.542 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T11:04:04.542 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T11:04:04.542 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T11:04:04.543 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_11 2023-10-11T11:04:04.642 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:Invalid command: unused arguments: ['mirror/image_11'] 2023-10-11T11:04:04.643 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd mirror status : get status for rbd mirror 2023-10-11T11:04:04.643 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:admin_socket: invalid command 2023-10-11T11:04:04.647 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-10-11T11:04:04.647 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_11 2023-10-11T11:04:04.647 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1:1 2023-10-11T11:04:04.648 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T11:04:04.648 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance 2023-10-11T11:04:04.648 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-10-11T11:04:04.648 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local val=cluster1:1 2023-10-11T11:04:04.649 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_var_name=cluster 2023-10-11T11:04:04.649 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance_var_name=instance 2023-10-11T11:04:04.649 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ cluster=cluster1 2023-10-11T11:04:04.649 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ instance=1 2023-10-11T11:04:04.650 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' 1 = cluster1:1 ']' 2023-10-11T11:04:04.650 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval cluster=cluster1 2023-10-11T11:04:04.650 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:04:04.650 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval instance=1 2023-10-11T11:04:04.651 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=1 2023-10-11T11:04:04.651 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-10-11T11:04:04.651 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local local_cluster=cluster1:1 2023-10-11T11:04:04.651 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster=cluster1 2023-10-11T11:04:04.652 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance 2023-10-11T11:04:04.652 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-10-11T11:04:04.652 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local val=cluster1:1 2023-10-11T11:04:04.652 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster_var_name=local_cluster 2023-10-11T11:04:04.653 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance_var_name=instance 2023-10-11T11:04:04.653 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:04:04.653 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=1 2023-10-11T11:04:04.653 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ '[' 1 = cluster1:1 ']' 2023-10-11T11:04:04.654 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval local_cluster=cluster1 2023-10-11T11:04:04.654 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ local_cluster=cluster1 2023-10-11T11:04:04.654 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval instance=1 2023-10-11T11:04:04.655 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ instance=1 2023-10-11T11:04:04.655 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-10-11T11:04:04.664 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T11:04:04.665 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T11:04:04.665 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T11:04:04.665 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_11 2023-10-11T11:04:04.761 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:Invalid command: unused arguments: ['mirror/image_11'] 2023-10-11T11:04:04.761 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd mirror status : get status for rbd mirror 2023-10-11T11:04:04.761 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:admin_socket: invalid command 2023-10-11T11:04:04.765 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-10-11T11:04:04.765 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_11 2023-10-11T11:04:04.765 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1:2 2023-10-11T11:04:04.766 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T11:04:04.766 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance 2023-10-11T11:04:04.766 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-10-11T11:04:04.767 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local val=cluster1:2 2023-10-11T11:04:04.767 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_var_name=cluster 2023-10-11T11:04:04.767 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance_var_name=instance 2023-10-11T11:04:04.767 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ cluster=cluster1 2023-10-11T11:04:04.768 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ instance=2 2023-10-11T11:04:04.768 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' 2 = cluster1:2 ']' 2023-10-11T11:04:04.768 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval cluster=cluster1 2023-10-11T11:04:04.768 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:04:04.769 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval instance=2 2023-10-11T11:04:04.769 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=2 2023-10-11T11:04:04.769 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-10-11T11:04:04.769 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local local_cluster=cluster1:2 2023-10-11T11:04:04.770 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster=cluster1 2023-10-11T11:04:04.770 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance 2023-10-11T11:04:04.770 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-10-11T11:04:04.771 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local val=cluster1:2 2023-10-11T11:04:04.771 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster_var_name=local_cluster 2023-10-11T11:04:04.771 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance_var_name=instance 2023-10-11T11:04:04.771 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:04:04.772 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=2 2023-10-11T11:04:04.772 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ '[' 2 = cluster1:2 ']' 2023-10-11T11:04:04.772 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval local_cluster=cluster1 2023-10-11T11:04:04.772 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ local_cluster=cluster1 2023-10-11T11:04:04.773 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval instance=2 2023-10-11T11:04:04.773 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ instance=2 2023-10-11T11:04:04.773 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-10-11T11:04:04.783 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-10-11T11:04:04.783 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-10-11T11:04:04.783 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-10-11T11:04:04.784 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_11 2023-10-11T11:04:04.929 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ return 0 2023-10-11T11:04:04.930 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ status_result=' "state": "Replaying"' 2023-10-11T11:04:04.930 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ echo ' "state": "Replaying"' 2023-10-11T11:04:04.930 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ grep -i Replaying 2023-10-11T11:04:04.931 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: "state": "Replaying" 2023-10-11T11:04:04.934 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ current_state=started 2023-10-11T11:04:04.935 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test started = started 2023-10-11T11:04:04.935 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:04:04.935 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_11 2023-10-11T11:04:04.935 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local local_cluster=cluster1 2023-10-11T11:04:04.936 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:04:04.936 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:04:04.936 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_11 2023-10-11T11:04:04.937 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' snapshot = journal ']' 2023-10-11T11:04:04.937 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' snapshot = snapshot ']' 2023-10-11T11:04:04.937 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_11 2023-10-11T11:04:04.937 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local local_cluster=cluster1 2023-10-11T11:04:04.938 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:04:04.938 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:04:04.938 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_11 2023-10-11T11:04:04.938 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ mkfname cluster2-mirror-image_11.status 2023-10-11T11:04:04.939 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ echo cluster2-mirror-image_11.status 2023-10-11T11:04:04.939 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -e 's|[/ ]|_|g' 2023-10-11T11:04:04.939 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local status_log=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_11.status 2023-10-11T11:04:04.939 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ mkfname cluster1-mirror-image_11.status 2023-10-11T11:04:04.940 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ echo cluster1-mirror-image_11.status 2023-10-11T11:04:04.940 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -e 's|[/ ]|_|g' 2023-10-11T11:04:04.940 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local local_status_log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_11.status 2023-10-11T11:04:04.940 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ mirror_image_snapshot cluster2 mirror image_11 2023-10-11T11:04:04.941 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:04:04.941 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:04:04.941 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_11 2023-10-11T11:04:04.941 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_11 2023-10-11T11:04:05.510 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:Snapshot ID: 139 2023-10-11T11:04:05.522 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_11 /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_11.status 2023-10-11T11:04:05.522 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:04:05.522 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:04:05.522 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_11 2023-10-11T11:04:05.523 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_11.status 2023-10-11T11:04:05.523 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_11 --format xml 2023-10-11T11:04:05.523 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:04:05.564 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-10-11T11:04:05.566 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snapshot_id=139 2023-10-11T11:04:05.566 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T11:04:05.567 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T11:04:05.567 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.2 2023-10-11T11:04:05.767 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_11 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_11.status 2023-10-11T11:04:05.768 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:04:05.768 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:04:05.768 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_11 2023-10-11T11:04:05.768 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_11.status 2023-10-11T11:04:05.769 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_11 --format xml 2023-10-11T11:04:05.769 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:04:05.813 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T11:04:05.815 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id=78 2023-10-11T11:04:05.815 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 139 = 78 2023-10-11T11:04:05.815 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T11:04:05.816 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.4 2023-10-11T11:04:06.216 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_11 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_11.status 2023-10-11T11:04:06.216 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:04:06.216 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:04:06.217 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_11 2023-10-11T11:04:06.217 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_11.status 2023-10-11T11:04:06.217 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_11 --format xml 2023-10-11T11:04:06.217 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:04:06.263 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T11:04:06.263 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T11:04:06.265 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:-:1.1: Document is empty 2023-10-11T11:04:06.265 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: 2023-10-11T11:04:06.265 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:^ 2023-10-11T11:04:06.266 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id= 2023-10-11T11:04:06.266 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 139 = '' 2023-10-11T11:04:06.266 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T11:04:06.266 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.8 2023-10-11T11:04:07.067 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_11 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_11.status 2023-10-11T11:04:07.067 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:04:07.067 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:04:07.068 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_11 2023-10-11T11:04:07.068 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_11.status 2023-10-11T11:04:07.068 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_11 --format xml 2023-10-11T11:04:07.068 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:04:07.130 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T11:04:07.130 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T11:04:07.131 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:-:1.1: Document is empty 2023-10-11T11:04:07.132 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: 2023-10-11T11:04:07.132 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:^ 2023-10-11T11:04:07.133 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id= 2023-10-11T11:04:07.133 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 139 = '' 2023-10-11T11:04:07.133 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T11:04:07.133 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 1.6 2023-10-11T11:04:08.733 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_11 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_11.status 2023-10-11T11:04:08.733 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:04:08.733 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:04:08.733 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_11 2023-10-11T11:04:08.734 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_11.status 2023-10-11T11:04:08.734 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_11 --format xml 2023-10-11T11:04:08.734 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:04:08.785 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T11:04:08.788 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id=139 2023-10-11T11:04:08.788 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 139 = 139 2023-10-11T11:04:08.788 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:04:08.789 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_snap_present cluster1 mirror image_11 snap 2023-10-11T11:04:08.789 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:04:08.789 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:04:08.789 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_11 2023-10-11T11:04:08.789 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap 2023-10-11T11:04:08.789 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T11:04:08.790 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-10-11T11:04:08.790 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 1 2023-10-11T11:04:09.789 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 info mirror/image_11@snap 2023-10-11T11:04:09.837 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:rbd image 'image_11': 2023-10-11T11:04:09.838 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: size 128 MiB in 32 objects 2023-10-11T11:04:09.838 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: order 22 (4 MiB objects) 2023-10-11T11:04:09.838 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: snapshot_count: 2 2023-10-11T11:04:09.838 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: id: 123d86515e24 2023-10-11T11:04:09.839 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: block_name_prefix: rbd_data.123d86515e24 2023-10-11T11:04:09.839 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: format: 2 2023-10-11T11:04:09.839 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-10-11T11:04:09.839 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: op_features: 2023-10-11T11:04:09.839 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: flags: 2023-10-11T11:04:09.840 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: create_timestamp: Wed Oct 11 10:55:45 2023 2023-10-11T11:04:09.840 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: access_timestamp: Wed Oct 11 10:55:45 2023 2023-10-11T11:04:09.840 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: modify_timestamp: Wed Oct 11 10:55:45 2023 2023-10-11T11:04:09.840 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: protected: False 2023-10-11T11:04:09.841 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring state: enabled 2023-10-11T11:04:09.841 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring mode: snapshot 2023-10-11T11:04:09.841 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring global id: 35668704-fbeb-4c06-998e-79dc6d9e7e60 2023-10-11T11:04:09.841 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring primary: false 2023-10-11T11:04:09.842 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:04:09.843 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ compare_image_snaps mirror image_11 snap 2023-10-11T11:04:09.843 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:04:09.843 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_11 2023-10-11T11:04:09.843 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap 2023-10-11T11:04:09.843 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local ret=0 2023-10-11T11:04:09.844 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local rmt_export=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_11.export 2023-10-11T11:04:09.844 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local loc_export=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_11.export 2023-10-11T11:04:09.844 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rm -f /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_11.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_11.export 2023-10-11T11:04:09.844 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 -p mirror export image_11@snap /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_11.export 2023-10-11T11:04:10.019 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-10-11T11:04:10.026 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 -p mirror export image_11@snap /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_11.export 2023-10-11T11:04:10.255 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-10-11T11:04:10.262 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ cmp /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_11.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_11.export 2023-10-11T11:04:10.316 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rm -f /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_11.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_11.export 2023-10-11T11:04:10.353 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:04:10.353 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T11:04:10.354 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_12 2023-10-11T11:04:10.354 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ create_snap cluster2 mirror image_12 snap 2023-10-11T11:04:10.354 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:04:10.354 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:04:10.355 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_12 2023-10-11T11:04:10.355 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap 2023-10-11T11:04:10.355 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 -p mirror snap create image_12@snap --debug-rbd=20 --debug-journaler=20 2023-10-11T11:04:12.398 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_replay_started cluster1 mirror image_12 2023-10-11T11:04:12.398 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:04:12.398 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:04:12.398 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_12 2023-10-11T11:04:12.399 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_replay_state cluster1 mirror image_12 started 2023-10-11T11:04:12.399 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:04:12.399 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:04:12.399 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_12 2023-10-11T11:04:12.399 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local state=started 2023-10-11T11:04:12.400 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T11:04:12.400 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-10-11T11:04:12.400 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 1 2023-10-11T11:04:13.400 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test_image_replay_state cluster1 mirror image_12 started 2023-10-11T11:04:13.400 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:04:13.400 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:04:13.400 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_12 2023-10-11T11:04:13.401 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local test_state=started 2023-10-11T11:04:13.401 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local status_result 2023-10-11T11:04:13.401 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_state=stopped 2023-10-11T11:04:13.401 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_12 2023-10-11T11:04:13.402 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_instance=cluster1 2023-10-11T11:04:13.402 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T11:04:13.402 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ grep -i state 2023-10-11T11:04:13.402 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:04:13.402 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance=cluster1 2023-10-11T11:04:13.403 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local loop_instance 2023-10-11T11:04:13.403 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-10-11T11:04:13.403 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sleep 0 2023-10-11T11:04:13.406 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-10-11T11:04:13.406 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ seq 0 3 2023-10-11T11:04:13.407 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-10-11T11:04:13.407 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_12 2023-10-11T11:04:13.407 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1:0 2023-10-11T11:04:13.407 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T11:04:13.408 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance 2023-10-11T11:04:13.408 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-10-11T11:04:13.408 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local val=cluster1:0 2023-10-11T11:04:13.408 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_var_name=cluster 2023-10-11T11:04:13.408 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance_var_name=instance 2023-10-11T11:04:13.409 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ cluster=cluster1 2023-10-11T11:04:13.409 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ instance=0 2023-10-11T11:04:13.409 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' 0 = cluster1:0 ']' 2023-10-11T11:04:13.409 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval cluster=cluster1 2023-10-11T11:04:13.409 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:04:13.410 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval instance=0 2023-10-11T11:04:13.410 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=0 2023-10-11T11:04:13.410 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-10-11T11:04:13.410 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local local_cluster=cluster1:0 2023-10-11T11:04:13.411 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster=cluster1 2023-10-11T11:04:13.411 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance 2023-10-11T11:04:13.411 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-10-11T11:04:13.411 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local val=cluster1:0 2023-10-11T11:04:13.411 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster_var_name=local_cluster 2023-10-11T11:04:13.412 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance_var_name=instance 2023-10-11T11:04:13.412 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:04:13.412 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=0 2023-10-11T11:04:13.412 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ '[' 0 = cluster1:0 ']' 2023-10-11T11:04:13.412 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval local_cluster=cluster1 2023-10-11T11:04:13.413 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ local_cluster=cluster1 2023-10-11T11:04:13.413 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval instance=0 2023-10-11T11:04:13.413 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ instance=0 2023-10-11T11:04:13.413 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-10-11T11:04:13.426 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T11:04:13.426 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T11:04:13.426 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T11:04:13.426 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_12 2023-10-11T11:04:13.527 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:Invalid command: unused arguments: ['mirror/image_12'] 2023-10-11T11:04:13.527 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd mirror status : get status for rbd mirror 2023-10-11T11:04:13.527 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:admin_socket: invalid command 2023-10-11T11:04:13.530 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-10-11T11:04:13.531 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_12 2023-10-11T11:04:13.531 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1:1 2023-10-11T11:04:13.531 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T11:04:13.531 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance 2023-10-11T11:04:13.532 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-10-11T11:04:13.532 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local val=cluster1:1 2023-10-11T11:04:13.532 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_var_name=cluster 2023-10-11T11:04:13.532 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance_var_name=instance 2023-10-11T11:04:13.533 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ cluster=cluster1 2023-10-11T11:04:13.533 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ instance=1 2023-10-11T11:04:13.533 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' 1 = cluster1:1 ']' 2023-10-11T11:04:13.533 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval cluster=cluster1 2023-10-11T11:04:13.534 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:04:13.534 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval instance=1 2023-10-11T11:04:13.534 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=1 2023-10-11T11:04:13.534 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-10-11T11:04:13.535 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local local_cluster=cluster1:1 2023-10-11T11:04:13.535 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster=cluster1 2023-10-11T11:04:13.535 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance 2023-10-11T11:04:13.535 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-10-11T11:04:13.536 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local val=cluster1:1 2023-10-11T11:04:13.536 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster_var_name=local_cluster 2023-10-11T11:04:13.536 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance_var_name=instance 2023-10-11T11:04:13.536 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:04:13.537 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=1 2023-10-11T11:04:13.537 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ '[' 1 = cluster1:1 ']' 2023-10-11T11:04:13.537 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval local_cluster=cluster1 2023-10-11T11:04:13.537 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ local_cluster=cluster1 2023-10-11T11:04:13.537 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval instance=1 2023-10-11T11:04:13.538 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ instance=1 2023-10-11T11:04:13.538 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-10-11T11:04:13.549 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T11:04:13.549 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T11:04:13.550 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T11:04:13.550 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_12 2023-10-11T11:04:13.643 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:Invalid command: unused arguments: ['mirror/image_12'] 2023-10-11T11:04:13.643 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd mirror status : get status for rbd mirror 2023-10-11T11:04:13.643 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:admin_socket: invalid command 2023-10-11T11:04:13.647 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-10-11T11:04:13.647 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_12 2023-10-11T11:04:13.647 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1:2 2023-10-11T11:04:13.648 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T11:04:13.648 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance 2023-10-11T11:04:13.648 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-10-11T11:04:13.649 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local val=cluster1:2 2023-10-11T11:04:13.649 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_var_name=cluster 2023-10-11T11:04:13.649 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance_var_name=instance 2023-10-11T11:04:13.649 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ cluster=cluster1 2023-10-11T11:04:13.650 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ instance=2 2023-10-11T11:04:13.650 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' 2 = cluster1:2 ']' 2023-10-11T11:04:13.650 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval cluster=cluster1 2023-10-11T11:04:13.650 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:04:13.651 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval instance=2 2023-10-11T11:04:13.651 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=2 2023-10-11T11:04:13.651 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-10-11T11:04:13.651 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local local_cluster=cluster1:2 2023-10-11T11:04:13.652 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster=cluster1 2023-10-11T11:04:13.652 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance 2023-10-11T11:04:13.652 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-10-11T11:04:13.652 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local val=cluster1:2 2023-10-11T11:04:13.653 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster_var_name=local_cluster 2023-10-11T11:04:13.653 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance_var_name=instance 2023-10-11T11:04:13.653 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:04:13.654 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=2 2023-10-11T11:04:13.654 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ '[' 2 = cluster1:2 ']' 2023-10-11T11:04:13.654 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval local_cluster=cluster1 2023-10-11T11:04:13.654 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ local_cluster=cluster1 2023-10-11T11:04:13.655 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval instance=2 2023-10-11T11:04:13.655 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ instance=2 2023-10-11T11:04:13.655 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-10-11T11:04:13.664 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-10-11T11:04:13.665 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-10-11T11:04:13.665 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-10-11T11:04:13.665 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_12 2023-10-11T11:04:13.810 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ return 0 2023-10-11T11:04:13.811 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ status_result=' "state": "Replaying"' 2023-10-11T11:04:13.812 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ echo ' "state": "Replaying"' 2023-10-11T11:04:13.812 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ grep -i Replaying 2023-10-11T11:04:13.813 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: "state": "Replaying" 2023-10-11T11:04:13.815 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ current_state=started 2023-10-11T11:04:13.815 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test started = started 2023-10-11T11:04:13.816 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:04:13.816 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_12 2023-10-11T11:04:13.816 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local local_cluster=cluster1 2023-10-11T11:04:13.816 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:04:13.817 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:04:13.817 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_12 2023-10-11T11:04:13.817 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' snapshot = journal ']' 2023-10-11T11:04:13.818 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' snapshot = snapshot ']' 2023-10-11T11:04:13.818 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_12 2023-10-11T11:04:13.818 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local local_cluster=cluster1 2023-10-11T11:04:13.818 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:04:13.819 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:04:13.819 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_12 2023-10-11T11:04:13.819 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ mkfname cluster2-mirror-image_12.status 2023-10-11T11:04:13.819 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ echo cluster2-mirror-image_12.status 2023-10-11T11:04:13.820 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -e 's|[/ ]|_|g' 2023-10-11T11:04:13.820 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local status_log=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_12.status 2023-10-11T11:04:13.820 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ mkfname cluster1-mirror-image_12.status 2023-10-11T11:04:13.820 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ echo cluster1-mirror-image_12.status 2023-10-11T11:04:13.820 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -e 's|[/ ]|_|g' 2023-10-11T11:04:13.822 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local local_status_log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_12.status 2023-10-11T11:04:13.822 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ mirror_image_snapshot cluster2 mirror image_12 2023-10-11T11:04:13.822 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:04:13.822 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:04:13.823 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_12 2023-10-11T11:04:13.823 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_12 2023-10-11T11:04:14.396 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:Snapshot ID: 141 2023-10-11T11:04:14.408 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_12 /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_12.status 2023-10-11T11:04:14.408 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:04:14.408 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:04:14.408 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_12 2023-10-11T11:04:14.408 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_12.status 2023-10-11T11:04:14.409 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_12 --format xml 2023-10-11T11:04:14.409 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:04:14.449 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-10-11T11:04:14.451 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snapshot_id=141 2023-10-11T11:04:14.451 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T11:04:14.451 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T11:04:14.452 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.2 2023-10-11T11:04:14.653 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_12 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_12.status 2023-10-11T11:04:14.653 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:04:14.653 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:04:14.653 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_12 2023-10-11T11:04:14.654 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_12.status 2023-10-11T11:04:14.654 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_12 --format xml 2023-10-11T11:04:14.654 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:04:14.698 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T11:04:14.699 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T11:04:14.700 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:-:1.1: Document is empty 2023-10-11T11:04:14.700 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: 2023-10-11T11:04:14.700 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:^ 2023-10-11T11:04:14.701 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id= 2023-10-11T11:04:14.701 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 141 = '' 2023-10-11T11:04:14.701 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T11:04:14.701 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.4 2023-10-11T11:04:15.102 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_12 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_12.status 2023-10-11T11:04:15.102 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:04:15.102 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:04:15.102 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_12 2023-10-11T11:04:15.103 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_12.status 2023-10-11T11:04:15.103 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_12 --format xml 2023-10-11T11:04:15.103 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:04:15.148 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T11:04:15.149 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T11:04:15.150 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:-:1.1: Document is empty 2023-10-11T11:04:15.150 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: 2023-10-11T11:04:15.150 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:^ 2023-10-11T11:04:15.151 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id= 2023-10-11T11:04:15.151 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 141 = '' 2023-10-11T11:04:15.151 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T11:04:15.151 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.8 2023-10-11T11:04:15.952 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_12 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_12.status 2023-10-11T11:04:15.952 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:04:15.952 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:04:15.952 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_12 2023-10-11T11:04:15.952 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_12.status 2023-10-11T11:04:15.953 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_12 --format xml 2023-10-11T11:04:15.953 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:04:15.997 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T11:04:15.999 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id=141 2023-10-11T11:04:16.000 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 141 = 141 2023-10-11T11:04:16.000 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:04:16.000 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_snap_present cluster1 mirror image_12 snap 2023-10-11T11:04:16.000 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:04:16.000 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:04:16.001 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_12 2023-10-11T11:04:16.001 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap 2023-10-11T11:04:16.001 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T11:04:16.001 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-10-11T11:04:16.001 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 1 2023-10-11T11:04:17.000 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 info mirror/image_12@snap 2023-10-11T11:04:17.047 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:rbd image 'image_12': 2023-10-11T11:04:17.047 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: size 128 MiB in 32 objects 2023-10-11T11:04:17.047 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: order 22 (4 MiB objects) 2023-10-11T11:04:17.047 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: snapshot_count: 2 2023-10-11T11:04:17.048 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: id: 123db8630789 2023-10-11T11:04:17.048 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: block_name_prefix: rbd_data.123db8630789 2023-10-11T11:04:17.048 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: format: 2 2023-10-11T11:04:17.048 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-10-11T11:04:17.048 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: op_features: 2023-10-11T11:04:17.048 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: flags: 2023-10-11T11:04:17.049 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: create_timestamp: Wed Oct 11 10:55:55 2023 2023-10-11T11:04:17.049 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: access_timestamp: Wed Oct 11 10:55:55 2023 2023-10-11T11:04:17.049 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: modify_timestamp: Wed Oct 11 10:55:55 2023 2023-10-11T11:04:17.049 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: protected: False 2023-10-11T11:04:17.049 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring state: enabled 2023-10-11T11:04:17.049 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring mode: snapshot 2023-10-11T11:04:17.050 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring global id: 418ab5b5-2cf6-4b23-a645-1106d804c432 2023-10-11T11:04:17.050 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring primary: false 2023-10-11T11:04:17.051 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:04:17.051 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ compare_image_snaps mirror image_12 snap 2023-10-11T11:04:17.051 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:04:17.051 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_12 2023-10-11T11:04:17.052 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap 2023-10-11T11:04:17.052 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local ret=0 2023-10-11T11:04:17.052 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local rmt_export=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_12.export 2023-10-11T11:04:17.052 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local loc_export=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_12.export 2023-10-11T11:04:17.052 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rm -f /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_12.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_12.export 2023-10-11T11:04:17.053 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 -p mirror export image_12@snap /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_12.export 2023-10-11T11:04:17.235 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-10-11T11:04:17.241 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 -p mirror export image_12@snap /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_12.export 2023-10-11T11:04:17.463 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-10-11T11:04:17.469 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ cmp /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_12.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_12.export 2023-10-11T11:04:17.523 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rm -f /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_12.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_12.export 2023-10-11T11:04:17.559 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:04:17.559 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T11:04:17.559 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_13 2023-10-11T11:04:17.560 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ create_snap cluster2 mirror image_13 snap 2023-10-11T11:04:17.560 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:04:17.560 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:04:17.560 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_13 2023-10-11T11:04:17.560 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap 2023-10-11T11:04:17.560 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 -p mirror snap create image_13@snap --debug-rbd=20 --debug-journaler=20 2023-10-11T11:04:19.483 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_replay_started cluster1 mirror image_13 2023-10-11T11:04:19.483 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:04:19.483 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:04:19.483 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_13 2023-10-11T11:04:19.484 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_replay_state cluster1 mirror image_13 started 2023-10-11T11:04:19.484 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:04:19.484 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:04:19.484 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_13 2023-10-11T11:04:19.485 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local state=started 2023-10-11T11:04:19.485 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T11:04:19.485 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-10-11T11:04:19.485 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 1 2023-10-11T11:04:20.483 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test_image_replay_state cluster1 mirror image_13 started 2023-10-11T11:04:20.484 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:04:20.484 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:04:20.484 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_13 2023-10-11T11:04:20.484 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local test_state=started 2023-10-11T11:04:20.484 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local status_result 2023-10-11T11:04:20.485 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_state=stopped 2023-10-11T11:04:20.486 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_13 2023-10-11T11:04:20.486 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_instance=cluster1 2023-10-11T11:04:20.486 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T11:04:20.486 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ grep -i state 2023-10-11T11:04:20.487 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:04:20.487 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance=cluster1 2023-10-11T11:04:20.487 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local loop_instance 2023-10-11T11:04:20.487 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-10-11T11:04:20.487 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sleep 0 2023-10-11T11:04:20.488 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-10-11T11:04:20.488 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ seq 0 3 2023-10-11T11:04:20.489 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-10-11T11:04:20.489 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_13 2023-10-11T11:04:20.490 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1:0 2023-10-11T11:04:20.490 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T11:04:20.490 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance 2023-10-11T11:04:20.490 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-10-11T11:04:20.490 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local val=cluster1:0 2023-10-11T11:04:20.490 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_var_name=cluster 2023-10-11T11:04:20.491 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance_var_name=instance 2023-10-11T11:04:20.491 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ cluster=cluster1 2023-10-11T11:04:20.491 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ instance=0 2023-10-11T11:04:20.491 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' 0 = cluster1:0 ']' 2023-10-11T11:04:20.491 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval cluster=cluster1 2023-10-11T11:04:20.492 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:04:20.492 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval instance=0 2023-10-11T11:04:20.493 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=0 2023-10-11T11:04:20.493 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-10-11T11:04:20.493 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local local_cluster=cluster1:0 2023-10-11T11:04:20.494 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster=cluster1 2023-10-11T11:04:20.494 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance 2023-10-11T11:04:20.494 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-10-11T11:04:20.494 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local val=cluster1:0 2023-10-11T11:04:20.494 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster_var_name=local_cluster 2023-10-11T11:04:20.495 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance_var_name=instance 2023-10-11T11:04:20.495 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:04:20.495 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=0 2023-10-11T11:04:20.495 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ '[' 0 = cluster1:0 ']' 2023-10-11T11:04:20.495 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval local_cluster=cluster1 2023-10-11T11:04:20.495 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ local_cluster=cluster1 2023-10-11T11:04:20.496 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval instance=0 2023-10-11T11:04:20.496 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ instance=0 2023-10-11T11:04:20.496 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-10-11T11:04:20.511 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T11:04:20.511 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T11:04:20.511 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T11:04:20.511 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_13 2023-10-11T11:04:20.611 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:Invalid command: unused arguments: ['mirror/image_13'] 2023-10-11T11:04:20.611 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd mirror status : get status for rbd mirror 2023-10-11T11:04:20.611 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:admin_socket: invalid command 2023-10-11T11:04:20.614 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-10-11T11:04:20.614 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_13 2023-10-11T11:04:20.614 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1:1 2023-10-11T11:04:20.614 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T11:04:20.615 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance 2023-10-11T11:04:20.615 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-10-11T11:04:20.615 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local val=cluster1:1 2023-10-11T11:04:20.615 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_var_name=cluster 2023-10-11T11:04:20.615 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance_var_name=instance 2023-10-11T11:04:20.616 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ cluster=cluster1 2023-10-11T11:04:20.616 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ instance=1 2023-10-11T11:04:20.616 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' 1 = cluster1:1 ']' 2023-10-11T11:04:20.616 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval cluster=cluster1 2023-10-11T11:04:20.616 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:04:20.616 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval instance=1 2023-10-11T11:04:20.617 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=1 2023-10-11T11:04:20.617 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-10-11T11:04:20.617 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local local_cluster=cluster1:1 2023-10-11T11:04:20.617 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster=cluster1 2023-10-11T11:04:20.617 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance 2023-10-11T11:04:20.617 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-10-11T11:04:20.618 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local val=cluster1:1 2023-10-11T11:04:20.618 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster_var_name=local_cluster 2023-10-11T11:04:20.618 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance_var_name=instance 2023-10-11T11:04:20.618 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:04:20.618 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=1 2023-10-11T11:04:20.619 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ '[' 1 = cluster1:1 ']' 2023-10-11T11:04:20.619 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval local_cluster=cluster1 2023-10-11T11:04:20.619 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ local_cluster=cluster1 2023-10-11T11:04:20.619 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval instance=1 2023-10-11T11:04:20.619 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ instance=1 2023-10-11T11:04:20.619 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-10-11T11:04:20.633 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T11:04:20.633 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T11:04:20.633 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T11:04:20.633 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_13 2023-10-11T11:04:20.727 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:Invalid command: unused arguments: ['mirror/image_13'] 2023-10-11T11:04:20.728 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd mirror status : get status for rbd mirror 2023-10-11T11:04:20.728 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:admin_socket: invalid command 2023-10-11T11:04:20.731 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-10-11T11:04:20.731 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_13 2023-10-11T11:04:20.731 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1:2 2023-10-11T11:04:20.731 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T11:04:20.731 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance 2023-10-11T11:04:20.732 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-10-11T11:04:20.732 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local val=cluster1:2 2023-10-11T11:04:20.732 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_var_name=cluster 2023-10-11T11:04:20.732 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance_var_name=instance 2023-10-11T11:04:20.732 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ cluster=cluster1 2023-10-11T11:04:20.733 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ instance=2 2023-10-11T11:04:20.733 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' 2 = cluster1:2 ']' 2023-10-11T11:04:20.733 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval cluster=cluster1 2023-10-11T11:04:20.733 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:04:20.733 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval instance=2 2023-10-11T11:04:20.733 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=2 2023-10-11T11:04:20.734 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-10-11T11:04:20.734 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local local_cluster=cluster1:2 2023-10-11T11:04:20.734 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster=cluster1 2023-10-11T11:04:20.734 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance 2023-10-11T11:04:20.734 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-10-11T11:04:20.734 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local val=cluster1:2 2023-10-11T11:04:20.735 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster_var_name=local_cluster 2023-10-11T11:04:20.735 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance_var_name=instance 2023-10-11T11:04:20.735 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:04:20.735 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=2 2023-10-11T11:04:20.735 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ '[' 2 = cluster1:2 ']' 2023-10-11T11:04:20.736 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval local_cluster=cluster1 2023-10-11T11:04:20.736 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ local_cluster=cluster1 2023-10-11T11:04:20.736 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval instance=2 2023-10-11T11:04:20.736 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ instance=2 2023-10-11T11:04:20.736 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-10-11T11:04:20.750 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-10-11T11:04:20.750 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-10-11T11:04:20.750 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-10-11T11:04:20.750 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_13 2023-10-11T11:04:20.895 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ return 0 2023-10-11T11:04:20.896 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ status_result=' "state": "Replaying"' 2023-10-11T11:04:20.896 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ echo ' "state": "Replaying"' 2023-10-11T11:04:20.897 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ grep -i Replaying 2023-10-11T11:04:20.897 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: "state": "Replaying" 2023-10-11T11:04:20.899 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ current_state=started 2023-10-11T11:04:20.899 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test started = started 2023-10-11T11:04:20.899 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:04:20.899 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_13 2023-10-11T11:04:20.899 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local local_cluster=cluster1 2023-10-11T11:04:20.900 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:04:20.900 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:04:20.900 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_13 2023-10-11T11:04:20.900 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' snapshot = journal ']' 2023-10-11T11:04:20.900 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' snapshot = snapshot ']' 2023-10-11T11:04:20.901 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_13 2023-10-11T11:04:20.901 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local local_cluster=cluster1 2023-10-11T11:04:20.901 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:04:20.901 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:04:20.901 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_13 2023-10-11T11:04:20.901 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ mkfname cluster2-mirror-image_13.status 2023-10-11T11:04:20.902 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ echo cluster2-mirror-image_13.status 2023-10-11T11:04:20.902 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -e 's|[/ ]|_|g' 2023-10-11T11:04:20.903 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local status_log=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_13.status 2023-10-11T11:04:20.903 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ mkfname cluster1-mirror-image_13.status 2023-10-11T11:04:20.903 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ echo cluster1-mirror-image_13.status 2023-10-11T11:04:20.903 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -e 's|[/ ]|_|g' 2023-10-11T11:04:20.904 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local local_status_log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_13.status 2023-10-11T11:04:20.904 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ mirror_image_snapshot cluster2 mirror image_13 2023-10-11T11:04:20.904 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:04:20.904 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:04:20.904 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_13 2023-10-11T11:04:20.905 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_13 2023-10-11T11:04:21.479 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:Snapshot ID: 143 2023-10-11T11:04:21.490 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_13 /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_13.status 2023-10-11T11:04:21.490 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:04:21.490 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:04:21.490 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_13 2023-10-11T11:04:21.491 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_13.status 2023-10-11T11:04:21.491 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_13 --format xml 2023-10-11T11:04:21.491 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:04:21.532 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-10-11T11:04:21.534 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snapshot_id=143 2023-10-11T11:04:21.534 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T11:04:21.535 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T11:04:21.535 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.2 2023-10-11T11:04:21.735 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_13 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_13.status 2023-10-11T11:04:21.735 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:04:21.736 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:04:21.736 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_13 2023-10-11T11:04:21.736 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_13.status 2023-10-11T11:04:21.736 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_13 --format xml 2023-10-11T11:04:21.737 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:04:21.782 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T11:04:21.782 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T11:04:21.783 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:-:1.1: Document is empty 2023-10-11T11:04:21.784 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: 2023-10-11T11:04:21.784 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:^ 2023-10-11T11:04:21.784 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id= 2023-10-11T11:04:21.785 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 143 = '' 2023-10-11T11:04:21.785 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T11:04:21.785 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.4 2023-10-11T11:04:22.185 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_13 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_13.status 2023-10-11T11:04:22.185 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:04:22.185 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:04:22.186 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_13 2023-10-11T11:04:22.186 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_13.status 2023-10-11T11:04:22.186 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_13 --format xml 2023-10-11T11:04:22.186 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:04:22.232 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T11:04:22.232 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T11:04:22.233 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:-:1.1: Document is empty 2023-10-11T11:04:22.234 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: 2023-10-11T11:04:22.234 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:^ 2023-10-11T11:04:22.234 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id= 2023-10-11T11:04:22.235 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 143 = '' 2023-10-11T11:04:22.235 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T11:04:22.235 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.8 2023-10-11T11:04:23.036 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_13 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_13.status 2023-10-11T11:04:23.036 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:04:23.036 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:04:23.037 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_13 2023-10-11T11:04:23.037 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_13.status 2023-10-11T11:04:23.037 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_13 --format xml 2023-10-11T11:04:23.037 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:04:23.081 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T11:04:23.083 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id=143 2023-10-11T11:04:23.083 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 143 = 143 2023-10-11T11:04:23.084 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:04:23.084 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_snap_present cluster1 mirror image_13 snap 2023-10-11T11:04:23.084 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:04:23.084 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:04:23.085 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_13 2023-10-11T11:04:23.085 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap 2023-10-11T11:04:23.085 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T11:04:23.086 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-10-11T11:04:23.086 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 1 2023-10-11T11:04:24.083 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 info mirror/image_13@snap 2023-10-11T11:04:24.131 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:rbd image 'image_13': 2023-10-11T11:04:24.131 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: size 128 MiB in 32 objects 2023-10-11T11:04:24.131 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: order 22 (4 MiB objects) 2023-10-11T11:04:24.131 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: snapshot_count: 2 2023-10-11T11:04:24.132 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: id: 123dbdda3dfa 2023-10-11T11:04:24.132 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: block_name_prefix: rbd_data.123dbdda3dfa 2023-10-11T11:04:24.132 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: format: 2 2023-10-11T11:04:24.132 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-10-11T11:04:24.132 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: op_features: 2023-10-11T11:04:24.133 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: flags: 2023-10-11T11:04:24.133 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: create_timestamp: Wed Oct 11 10:56:08 2023 2023-10-11T11:04:24.133 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: access_timestamp: Wed Oct 11 10:56:08 2023 2023-10-11T11:04:24.133 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: modify_timestamp: Wed Oct 11 10:56:08 2023 2023-10-11T11:04:24.133 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: protected: False 2023-10-11T11:04:24.133 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring state: enabled 2023-10-11T11:04:24.134 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring mode: snapshot 2023-10-11T11:04:24.134 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring global id: bea67517-2558-4f8d-8c9b-71c5be4fe773 2023-10-11T11:04:24.134 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring primary: false 2023-10-11T11:04:24.135 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:04:24.135 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ compare_image_snaps mirror image_13 snap 2023-10-11T11:04:24.136 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:04:24.136 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_13 2023-10-11T11:04:24.136 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap 2023-10-11T11:04:24.136 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local ret=0 2023-10-11T11:04:24.136 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local rmt_export=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_13.export 2023-10-11T11:04:24.137 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local loc_export=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_13.export 2023-10-11T11:04:24.137 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rm -f /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_13.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_13.export 2023-10-11T11:04:24.137 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 -p mirror export image_13@snap /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_13.export 2023-10-11T11:04:24.315 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-10-11T11:04:24.321 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 -p mirror export image_13@snap /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_13.export 2023-10-11T11:04:24.580 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-10-11T11:04:24.586 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ cmp /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_13.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_13.export 2023-10-11T11:04:24.641 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rm -f /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_13.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_13.export 2023-10-11T11:04:24.677 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:04:24.677 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T11:04:24.678 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_14 2023-10-11T11:04:24.678 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ create_snap cluster2 mirror image_14 snap 2023-10-11T11:04:24.678 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:04:24.678 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:04:24.678 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_14 2023-10-11T11:04:24.679 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap 2023-10-11T11:04:24.679 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 -p mirror snap create image_14@snap --debug-rbd=20 --debug-journaler=20 2023-10-11T11:04:26.496 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_replay_started cluster1 mirror image_14 2023-10-11T11:04:26.496 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:04:26.496 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:04:26.497 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_14 2023-10-11T11:04:26.497 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_replay_state cluster1 mirror image_14 started 2023-10-11T11:04:26.497 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:04:26.497 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:04:26.497 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_14 2023-10-11T11:04:26.497 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local state=started 2023-10-11T11:04:26.498 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T11:04:26.498 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-10-11T11:04:26.498 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 1 2023-10-11T11:04:27.497 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test_image_replay_state cluster1 mirror image_14 started 2023-10-11T11:04:27.497 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:04:27.498 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:04:27.498 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_14 2023-10-11T11:04:27.498 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local test_state=started 2023-10-11T11:04:27.498 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local status_result 2023-10-11T11:04:27.498 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_state=stopped 2023-10-11T11:04:27.499 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_14 2023-10-11T11:04:27.500 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ grep -i state 2023-10-11T11:04:27.500 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_instance=cluster1 2023-10-11T11:04:27.500 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T11:04:27.500 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:04:27.500 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance=cluster1 2023-10-11T11:04:27.501 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local loop_instance 2023-10-11T11:04:27.501 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-10-11T11:04:27.501 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sleep 0 2023-10-11T11:04:27.501 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-10-11T11:04:27.501 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ seq 0 3 2023-10-11T11:04:27.504 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-10-11T11:04:27.504 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_14 2023-10-11T11:04:27.504 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1:0 2023-10-11T11:04:27.505 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T11:04:27.505 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance 2023-10-11T11:04:27.505 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-10-11T11:04:27.505 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local val=cluster1:0 2023-10-11T11:04:27.505 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_var_name=cluster 2023-10-11T11:04:27.505 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance_var_name=instance 2023-10-11T11:04:27.506 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ cluster=cluster1 2023-10-11T11:04:27.506 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ instance=0 2023-10-11T11:04:27.506 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' 0 = cluster1:0 ']' 2023-10-11T11:04:27.506 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval cluster=cluster1 2023-10-11T11:04:27.507 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:04:27.507 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval instance=0 2023-10-11T11:04:27.507 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=0 2023-10-11T11:04:27.507 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-10-11T11:04:27.507 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local local_cluster=cluster1:0 2023-10-11T11:04:27.508 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster=cluster1 2023-10-11T11:04:27.508 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance 2023-10-11T11:04:27.508 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-10-11T11:04:27.508 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local val=cluster1:0 2023-10-11T11:04:27.508 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster_var_name=local_cluster 2023-10-11T11:04:27.508 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance_var_name=instance 2023-10-11T11:04:27.509 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:04:27.509 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=0 2023-10-11T11:04:27.509 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ '[' 0 = cluster1:0 ']' 2023-10-11T11:04:27.509 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval local_cluster=cluster1 2023-10-11T11:04:27.509 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ local_cluster=cluster1 2023-10-11T11:04:27.509 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval instance=0 2023-10-11T11:04:27.510 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ instance=0 2023-10-11T11:04:27.510 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-10-11T11:04:27.524 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T11:04:27.524 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T11:04:27.524 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T11:04:27.525 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_14 2023-10-11T11:04:27.625 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:Invalid command: unused arguments: ['mirror/image_14'] 2023-10-11T11:04:27.625 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd mirror status : get status for rbd mirror 2023-10-11T11:04:27.625 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:admin_socket: invalid command 2023-10-11T11:04:27.627 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-10-11T11:04:27.628 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_14 2023-10-11T11:04:27.628 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1:1 2023-10-11T11:04:27.628 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T11:04:27.628 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance 2023-10-11T11:04:27.628 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-10-11T11:04:27.629 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local val=cluster1:1 2023-10-11T11:04:27.629 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_var_name=cluster 2023-10-11T11:04:27.629 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance_var_name=instance 2023-10-11T11:04:27.629 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ cluster=cluster1 2023-10-11T11:04:27.629 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ instance=1 2023-10-11T11:04:27.629 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' 1 = cluster1:1 ']' 2023-10-11T11:04:27.630 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval cluster=cluster1 2023-10-11T11:04:27.630 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:04:27.630 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval instance=1 2023-10-11T11:04:27.630 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=1 2023-10-11T11:04:27.630 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-10-11T11:04:27.631 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local local_cluster=cluster1:1 2023-10-11T11:04:27.631 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster=cluster1 2023-10-11T11:04:27.631 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance 2023-10-11T11:04:27.631 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-10-11T11:04:27.631 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local val=cluster1:1 2023-10-11T11:04:27.631 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster_var_name=local_cluster 2023-10-11T11:04:27.632 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance_var_name=instance 2023-10-11T11:04:27.632 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:04:27.632 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=1 2023-10-11T11:04:27.632 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ '[' 1 = cluster1:1 ']' 2023-10-11T11:04:27.632 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval local_cluster=cluster1 2023-10-11T11:04:27.632 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ local_cluster=cluster1 2023-10-11T11:04:27.633 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval instance=1 2023-10-11T11:04:27.633 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ instance=1 2023-10-11T11:04:27.633 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-10-11T11:04:27.647 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T11:04:27.647 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T11:04:27.647 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T11:04:27.647 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_14 2023-10-11T11:04:27.742 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:Invalid command: unused arguments: ['mirror/image_14'] 2023-10-11T11:04:27.742 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd mirror status : get status for rbd mirror 2023-10-11T11:04:27.742 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:admin_socket: invalid command 2023-10-11T11:04:27.745 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-10-11T11:04:27.745 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_14 2023-10-11T11:04:27.745 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1:2 2023-10-11T11:04:27.746 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T11:04:27.746 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance 2023-10-11T11:04:27.746 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-10-11T11:04:27.746 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local val=cluster1:2 2023-10-11T11:04:27.746 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_var_name=cluster 2023-10-11T11:04:27.747 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance_var_name=instance 2023-10-11T11:04:27.747 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ cluster=cluster1 2023-10-11T11:04:27.747 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ instance=2 2023-10-11T11:04:27.747 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' 2 = cluster1:2 ']' 2023-10-11T11:04:27.747 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval cluster=cluster1 2023-10-11T11:04:27.747 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:04:27.748 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval instance=2 2023-10-11T11:04:27.748 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=2 2023-10-11T11:04:27.748 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-10-11T11:04:27.748 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local local_cluster=cluster1:2 2023-10-11T11:04:27.748 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster=cluster1 2023-10-11T11:04:27.749 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance 2023-10-11T11:04:27.749 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-10-11T11:04:27.749 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local val=cluster1:2 2023-10-11T11:04:27.749 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster_var_name=local_cluster 2023-10-11T11:04:27.750 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance_var_name=instance 2023-10-11T11:04:27.750 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:04:27.750 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=2 2023-10-11T11:04:27.750 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ '[' 2 = cluster1:2 ']' 2023-10-11T11:04:27.750 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval local_cluster=cluster1 2023-10-11T11:04:27.751 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ local_cluster=cluster1 2023-10-11T11:04:27.751 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval instance=2 2023-10-11T11:04:27.751 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ instance=2 2023-10-11T11:04:27.751 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-10-11T11:04:27.764 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-10-11T11:04:27.764 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-10-11T11:04:27.765 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-10-11T11:04:27.765 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_14 2023-10-11T11:04:27.912 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ return 0 2023-10-11T11:04:27.913 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ status_result=' "state": "Replaying"' 2023-10-11T11:04:27.913 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ echo ' "state": "Replaying"' 2023-10-11T11:04:27.914 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ grep -i Replaying 2023-10-11T11:04:27.914 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: "state": "Replaying" 2023-10-11T11:04:27.916 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ current_state=started 2023-10-11T11:04:27.917 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test started = started 2023-10-11T11:04:27.917 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:04:27.917 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_14 2023-10-11T11:04:27.917 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local local_cluster=cluster1 2023-10-11T11:04:27.917 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:04:27.918 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:04:27.918 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_14 2023-10-11T11:04:27.918 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' snapshot = journal ']' 2023-10-11T11:04:27.918 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' snapshot = snapshot ']' 2023-10-11T11:04:27.918 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_14 2023-10-11T11:04:27.919 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local local_cluster=cluster1 2023-10-11T11:04:27.919 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:04:27.919 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:04:27.919 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_14 2023-10-11T11:04:27.919 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ mkfname cluster2-mirror-image_14.status 2023-10-11T11:04:27.920 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ echo cluster2-mirror-image_14.status 2023-10-11T11:04:27.920 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -e 's|[/ ]|_|g' 2023-10-11T11:04:27.920 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local status_log=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_14.status 2023-10-11T11:04:27.920 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ mkfname cluster1-mirror-image_14.status 2023-10-11T11:04:27.920 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ echo cluster1-mirror-image_14.status 2023-10-11T11:04:27.921 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -e 's|[/ ]|_|g' 2023-10-11T11:04:27.921 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local local_status_log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_14.status 2023-10-11T11:04:27.922 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ mirror_image_snapshot cluster2 mirror image_14 2023-10-11T11:04:27.922 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:04:27.922 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:04:27.922 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_14 2023-10-11T11:04:27.923 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_14 2023-10-11T11:04:28.503 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:Snapshot ID: 145 2023-10-11T11:04:28.514 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_14 /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_14.status 2023-10-11T11:04:28.515 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:04:28.515 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:04:28.515 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_14 2023-10-11T11:04:28.516 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_14.status 2023-10-11T11:04:28.516 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_14 --format xml 2023-10-11T11:04:28.516 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:04:28.558 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-10-11T11:04:28.560 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snapshot_id=145 2023-10-11T11:04:28.560 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T11:04:28.561 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T11:04:28.561 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.2 2023-10-11T11:04:28.762 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_14 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_14.status 2023-10-11T11:04:28.762 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:04:28.762 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:04:28.762 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_14 2023-10-11T11:04:28.763 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_14.status 2023-10-11T11:04:28.763 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_14 --format xml 2023-10-11T11:04:28.763 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:04:28.808 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T11:04:28.808 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T11:04:28.809 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:-:1.1: Document is empty 2023-10-11T11:04:28.809 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: 2023-10-11T11:04:28.810 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:^ 2023-10-11T11:04:28.811 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id= 2023-10-11T11:04:28.811 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 145 = '' 2023-10-11T11:04:28.811 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T11:04:28.811 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.4 2023-10-11T11:04:29.211 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_14 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_14.status 2023-10-11T11:04:29.211 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:04:29.211 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:04:29.212 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_14 2023-10-11T11:04:29.212 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_14.status 2023-10-11T11:04:29.212 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_14 --format xml 2023-10-11T11:04:29.212 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:04:29.259 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T11:04:29.260 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T11:04:29.261 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:-:1.1: Document is empty 2023-10-11T11:04:29.261 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: 2023-10-11T11:04:29.261 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:^ 2023-10-11T11:04:29.262 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id= 2023-10-11T11:04:29.262 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 145 = '' 2023-10-11T11:04:29.263 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T11:04:29.263 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.8 2023-10-11T11:04:29.698 INFO:tasks.ceph.cluster2.osd.1.smithi111.stderr:2023-10-11T11:04:29.694+0000 7f9628714700 -1 reset not still connected to 0x56424a0672b0 2023-10-11T11:04:29.698 INFO:tasks.ceph.cluster2.osd.1.smithi111.stderr:2023-10-11T11:04:29.694+0000 7f9628714700 -1 reset not still connected to 0x564256b30270 2023-10-11T11:04:29.698 INFO:tasks.ceph.cluster2.osd.1.smithi111.stderr:2023-10-11T11:04:29.694+0000 7f9628714700 -1 reset not still connected to 0x564256b30a90 2023-10-11T11:04:29.699 INFO:tasks.ceph.cluster2.osd.1.smithi111.stderr:2023-10-11T11:04:29.694+0000 7f9628714700 -1 reset not still connected to 0x564259783040 2023-10-11T11:04:29.699 INFO:tasks.ceph.cluster2.osd.1.smithi111.stderr:2023-10-11T11:04:29.694+0000 7f9628714700 -1 reset not still connected to 0x56427441d520 2023-10-11T11:04:29.699 INFO:tasks.ceph.cluster2.osd.1.smithi111.stderr:2023-10-11T11:04:29.694+0000 7f9628714700 -1 reset not still connected to 0x564277507860 2023-10-11T11:04:29.699 INFO:tasks.ceph.cluster2.osd.1.smithi111.stderr:2023-10-11T11:04:29.694+0000 7f9628714700 -1 reset not still connected to 0x564277e4aea0 2023-10-11T11:04:29.700 INFO:tasks.ceph.cluster2.osd.1.smithi111.stderr:2023-10-11T11:04:29.694+0000 7f9628714700 -1 reset not still connected to 0x5642822b8ea0 2023-10-11T11:04:29.700 INFO:tasks.ceph.cluster2.osd.1.smithi111.stderr:2023-10-11T11:04:29.694+0000 7f9628714700 -1 reset not still connected to 0x564288a2a1a0 2023-10-11T11:04:30.062 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_14 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_14.status 2023-10-11T11:04:30.063 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:04:30.063 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:04:30.063 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_14 2023-10-11T11:04:30.063 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_14.status 2023-10-11T11:04:30.064 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_14 --format xml 2023-10-11T11:04:30.064 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:04:30.107 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T11:04:30.109 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id=145 2023-10-11T11:04:30.109 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 145 = 145 2023-10-11T11:04:30.109 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:04:30.110 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_snap_present cluster1 mirror image_14 snap 2023-10-11T11:04:30.110 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:04:30.110 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:04:30.110 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_14 2023-10-11T11:04:30.110 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap 2023-10-11T11:04:30.111 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T11:04:30.111 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-10-11T11:04:30.111 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 1 2023-10-11T11:04:31.110 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 info mirror/image_14@snap 2023-10-11T11:04:31.152 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:rbd image 'image_14': 2023-10-11T11:04:31.152 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: size 128 MiB in 32 objects 2023-10-11T11:04:31.153 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: order 22 (4 MiB objects) 2023-10-11T11:04:31.153 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: snapshot_count: 2 2023-10-11T11:04:31.153 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: id: 123de76b19c5 2023-10-11T11:04:31.153 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: block_name_prefix: rbd_data.123de76b19c5 2023-10-11T11:04:31.153 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: format: 2 2023-10-11T11:04:31.154 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-10-11T11:04:31.154 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: op_features: 2023-10-11T11:04:31.154 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: flags: 2023-10-11T11:04:31.154 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: create_timestamp: Wed Oct 11 10:56:16 2023 2023-10-11T11:04:31.154 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: access_timestamp: Wed Oct 11 10:56:16 2023 2023-10-11T11:04:31.154 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: modify_timestamp: Wed Oct 11 10:56:16 2023 2023-10-11T11:04:31.155 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: protected: False 2023-10-11T11:04:31.155 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring state: enabled 2023-10-11T11:04:31.155 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring mode: snapshot 2023-10-11T11:04:31.155 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring global id: cc29ef48-4a54-4049-9722-bc6c6b7caf39 2023-10-11T11:04:31.155 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring primary: false 2023-10-11T11:04:31.157 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:04:31.157 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ compare_image_snaps mirror image_14 snap 2023-10-11T11:04:31.157 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:04:31.157 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_14 2023-10-11T11:04:31.157 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap 2023-10-11T11:04:31.158 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local ret=0 2023-10-11T11:04:31.158 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local rmt_export=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_14.export 2023-10-11T11:04:31.158 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local loc_export=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_14.export 2023-10-11T11:04:31.158 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rm -f /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_14.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_14.export 2023-10-11T11:04:31.158 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 -p mirror export image_14@snap /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_14.export 2023-10-11T11:04:31.332 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-10-11T11:04:31.338 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 -p mirror export image_14@snap /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_14.export 2023-10-11T11:04:31.577 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-10-11T11:04:31.583 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ cmp /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_14.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_14.export 2023-10-11T11:04:31.638 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rm -f /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_14.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_14.export 2023-10-11T11:04:31.674 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:04:31.674 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T11:04:31.675 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_15 2023-10-11T11:04:31.675 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ create_snap cluster2 mirror image_15 snap 2023-10-11T11:04:31.675 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:04:31.675 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:04:31.675 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_15 2023-10-11T11:04:31.676 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap 2023-10-11T11:04:31.676 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 -p mirror snap create image_15@snap --debug-rbd=20 --debug-journaler=20 2023-10-11T11:04:33.511 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_replay_started cluster1 mirror image_15 2023-10-11T11:04:33.512 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:04:33.512 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:04:33.512 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_15 2023-10-11T11:04:33.513 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_replay_state cluster1 mirror image_15 started 2023-10-11T11:04:33.513 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:04:33.513 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:04:33.513 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_15 2023-10-11T11:04:33.513 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local state=started 2023-10-11T11:04:33.514 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T11:04:33.514 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-10-11T11:04:33.514 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 1 2023-10-11T11:04:34.514 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test_image_replay_state cluster1 mirror image_15 started 2023-10-11T11:04:34.514 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:04:34.515 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:04:34.515 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_15 2023-10-11T11:04:34.515 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local test_state=started 2023-10-11T11:04:34.515 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local status_result 2023-10-11T11:04:34.516 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_state=stopped 2023-10-11T11:04:34.516 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_15 2023-10-11T11:04:34.516 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_instance=cluster1 2023-10-11T11:04:34.517 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T11:04:34.517 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ grep -i state 2023-10-11T11:04:34.517 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:04:34.517 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance=cluster1 2023-10-11T11:04:34.518 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local loop_instance 2023-10-11T11:04:34.518 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-10-11T11:04:34.518 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sleep 0 2023-10-11T11:04:34.519 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-10-11T11:04:34.522 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ seq 0 3 2023-10-11T11:04:34.523 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-10-11T11:04:34.523 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_15 2023-10-11T11:04:34.523 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1:0 2023-10-11T11:04:34.524 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T11:04:34.524 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance 2023-10-11T11:04:34.524 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-10-11T11:04:34.524 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local val=cluster1:0 2023-10-11T11:04:34.525 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_var_name=cluster 2023-10-11T11:04:34.525 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance_var_name=instance 2023-10-11T11:04:34.525 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ cluster=cluster1 2023-10-11T11:04:34.525 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ instance=0 2023-10-11T11:04:34.526 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' 0 = cluster1:0 ']' 2023-10-11T11:04:34.526 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval cluster=cluster1 2023-10-11T11:04:34.526 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:04:34.527 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval instance=0 2023-10-11T11:04:34.527 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=0 2023-10-11T11:04:34.527 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-10-11T11:04:34.527 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local local_cluster=cluster1:0 2023-10-11T11:04:34.528 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster=cluster1 2023-10-11T11:04:34.528 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance 2023-10-11T11:04:34.528 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-10-11T11:04:34.528 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local val=cluster1:0 2023-10-11T11:04:34.529 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster_var_name=local_cluster 2023-10-11T11:04:34.529 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance_var_name=instance 2023-10-11T11:04:34.529 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:04:34.529 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=0 2023-10-11T11:04:34.530 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ '[' 0 = cluster1:0 ']' 2023-10-11T11:04:34.530 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval local_cluster=cluster1 2023-10-11T11:04:34.530 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ local_cluster=cluster1 2023-10-11T11:04:34.531 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval instance=0 2023-10-11T11:04:34.531 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ instance=0 2023-10-11T11:04:34.531 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-10-11T11:04:34.539 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T11:04:34.540 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T11:04:34.540 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T11:04:34.540 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_15 2023-10-11T11:04:34.640 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:Invalid command: unused arguments: ['mirror/image_15'] 2023-10-11T11:04:34.641 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd mirror status : get status for rbd mirror 2023-10-11T11:04:34.641 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:admin_socket: invalid command 2023-10-11T11:04:34.644 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-10-11T11:04:34.645 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_15 2023-10-11T11:04:34.645 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1:1 2023-10-11T11:04:34.645 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T11:04:34.646 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance 2023-10-11T11:04:34.646 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-10-11T11:04:34.646 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local val=cluster1:1 2023-10-11T11:04:34.646 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_var_name=cluster 2023-10-11T11:04:34.647 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance_var_name=instance 2023-10-11T11:04:34.647 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ cluster=cluster1 2023-10-11T11:04:34.647 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ instance=1 2023-10-11T11:04:34.647 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' 1 = cluster1:1 ']' 2023-10-11T11:04:34.648 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval cluster=cluster1 2023-10-11T11:04:34.648 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:04:34.648 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval instance=1 2023-10-11T11:04:34.648 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=1 2023-10-11T11:04:34.649 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-10-11T11:04:34.649 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local local_cluster=cluster1:1 2023-10-11T11:04:34.649 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster=cluster1 2023-10-11T11:04:34.649 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance 2023-10-11T11:04:34.650 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-10-11T11:04:34.650 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local val=cluster1:1 2023-10-11T11:04:34.650 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster_var_name=local_cluster 2023-10-11T11:04:34.651 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance_var_name=instance 2023-10-11T11:04:34.651 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:04:34.651 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=1 2023-10-11T11:04:34.651 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ '[' 1 = cluster1:1 ']' 2023-10-11T11:04:34.652 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval local_cluster=cluster1 2023-10-11T11:04:34.652 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ local_cluster=cluster1 2023-10-11T11:04:34.652 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval instance=1 2023-10-11T11:04:34.652 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ instance=1 2023-10-11T11:04:34.653 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-10-11T11:04:34.662 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T11:04:34.662 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T11:04:34.663 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T11:04:34.663 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_15 2023-10-11T11:04:34.758 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:Invalid command: unused arguments: ['mirror/image_15'] 2023-10-11T11:04:34.758 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd mirror status : get status for rbd mirror 2023-10-11T11:04:34.758 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:admin_socket: invalid command 2023-10-11T11:04:34.761 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-10-11T11:04:34.762 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_15 2023-10-11T11:04:34.762 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1:2 2023-10-11T11:04:34.762 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T11:04:34.762 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance 2023-10-11T11:04:34.763 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-10-11T11:04:34.763 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local val=cluster1:2 2023-10-11T11:04:34.763 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_var_name=cluster 2023-10-11T11:04:34.763 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance_var_name=instance 2023-10-11T11:04:34.763 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ cluster=cluster1 2023-10-11T11:04:34.764 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ instance=2 2023-10-11T11:04:34.764 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' 2 = cluster1:2 ']' 2023-10-11T11:04:34.764 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval cluster=cluster1 2023-10-11T11:04:34.764 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:04:34.765 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval instance=2 2023-10-11T11:04:34.765 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=2 2023-10-11T11:04:34.765 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-10-11T11:04:34.765 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local local_cluster=cluster1:2 2023-10-11T11:04:34.766 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster=cluster1 2023-10-11T11:04:34.766 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance 2023-10-11T11:04:34.766 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-10-11T11:04:34.766 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local val=cluster1:2 2023-10-11T11:04:34.767 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster_var_name=local_cluster 2023-10-11T11:04:34.767 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance_var_name=instance 2023-10-11T11:04:34.767 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:04:34.767 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=2 2023-10-11T11:04:34.767 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ '[' 2 = cluster1:2 ']' 2023-10-11T11:04:34.768 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval local_cluster=cluster1 2023-10-11T11:04:34.768 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ local_cluster=cluster1 2023-10-11T11:04:34.768 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval instance=2 2023-10-11T11:04:34.768 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ instance=2 2023-10-11T11:04:34.769 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-10-11T11:04:34.780 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-10-11T11:04:34.780 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-10-11T11:04:34.780 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-10-11T11:04:34.781 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_15 2023-10-11T11:04:34.926 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ return 0 2023-10-11T11:04:34.926 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ status_result=' "state": "Replaying"' 2023-10-11T11:04:34.927 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ echo ' "state": "Replaying"' 2023-10-11T11:04:34.927 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ grep -i Replaying 2023-10-11T11:04:34.927 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: "state": "Replaying" 2023-10-11T11:04:34.929 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ current_state=started 2023-10-11T11:04:34.930 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test started = started 2023-10-11T11:04:34.930 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:04:34.930 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_15 2023-10-11T11:04:34.930 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local local_cluster=cluster1 2023-10-11T11:04:34.931 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:04:34.931 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:04:34.931 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_15 2023-10-11T11:04:34.931 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' snapshot = journal ']' 2023-10-11T11:04:34.931 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' snapshot = snapshot ']' 2023-10-11T11:04:34.932 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_15 2023-10-11T11:04:34.932 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local local_cluster=cluster1 2023-10-11T11:04:34.932 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:04:34.932 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:04:34.932 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_15 2023-10-11T11:04:34.932 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ mkfname cluster2-mirror-image_15.status 2023-10-11T11:04:34.933 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ echo cluster2-mirror-image_15.status 2023-10-11T11:04:34.933 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -e 's|[/ ]|_|g' 2023-10-11T11:04:34.933 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local status_log=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_15.status 2023-10-11T11:04:34.933 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ mkfname cluster1-mirror-image_15.status 2023-10-11T11:04:34.934 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ echo cluster1-mirror-image_15.status 2023-10-11T11:04:34.934 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -e 's|[/ ]|_|g' 2023-10-11T11:04:34.935 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local local_status_log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_15.status 2023-10-11T11:04:34.935 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ mirror_image_snapshot cluster2 mirror image_15 2023-10-11T11:04:34.935 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:04:34.935 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:04:34.935 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_15 2023-10-11T11:04:34.936 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_15 2023-10-11T11:04:35.518 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:Snapshot ID: 147 2023-10-11T11:04:35.530 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_15 /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_15.status 2023-10-11T11:04:35.530 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:04:35.530 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:04:35.531 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_15 2023-10-11T11:04:35.531 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_15.status 2023-10-11T11:04:35.531 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_15 --format xml 2023-10-11T11:04:35.531 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:04:35.574 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-10-11T11:04:35.576 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snapshot_id=147 2023-10-11T11:04:35.576 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T11:04:35.576 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T11:04:35.576 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.2 2023-10-11T11:04:35.777 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_15 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_15.status 2023-10-11T11:04:35.777 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:04:35.777 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:04:35.778 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_15 2023-10-11T11:04:35.778 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_15.status 2023-10-11T11:04:35.778 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_15 --format xml 2023-10-11T11:04:35.778 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:04:35.824 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T11:04:35.826 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id=82 2023-10-11T11:04:35.826 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 147 = 82 2023-10-11T11:04:35.826 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T11:04:35.826 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.4 2023-10-11T11:04:36.227 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_15 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_15.status 2023-10-11T11:04:36.228 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:04:36.228 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:04:36.228 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_15 2023-10-11T11:04:36.228 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_15.status 2023-10-11T11:04:36.228 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_15 --format xml 2023-10-11T11:04:36.229 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:04:36.273 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T11:04:36.274 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T11:04:36.275 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:-:1.1: Document is empty 2023-10-11T11:04:36.275 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: 2023-10-11T11:04:36.276 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:^ 2023-10-11T11:04:36.276 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id= 2023-10-11T11:04:36.277 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 147 = '' 2023-10-11T11:04:36.277 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T11:04:36.277 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.8 2023-10-11T11:04:37.077 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_15 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_15.status 2023-10-11T11:04:37.078 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:04:37.078 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:04:37.078 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_15 2023-10-11T11:04:37.078 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_15.status 2023-10-11T11:04:37.078 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_15 --format xml 2023-10-11T11:04:37.078 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:04:37.148 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T11:04:37.148 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T11:04:37.149 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:-:1.1: Document is empty 2023-10-11T11:04:37.150 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: 2023-10-11T11:04:37.150 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:^ 2023-10-11T11:04:37.151 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id= 2023-10-11T11:04:37.151 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 147 = '' 2023-10-11T11:04:37.151 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T11:04:37.151 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 1.6 2023-10-11T11:04:38.751 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_15 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_15.status 2023-10-11T11:04:38.751 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:04:38.752 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:04:38.752 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_15 2023-10-11T11:04:38.752 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_15.status 2023-10-11T11:04:38.752 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_15 --format xml 2023-10-11T11:04:38.752 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:04:38.803 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T11:04:38.806 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id=147 2023-10-11T11:04:38.806 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 147 = 147 2023-10-11T11:04:38.806 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:04:38.807 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_snap_present cluster1 mirror image_15 snap 2023-10-11T11:04:38.807 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:04:38.807 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:04:38.807 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_15 2023-10-11T11:04:38.807 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap 2023-10-11T11:04:38.808 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T11:04:38.808 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-10-11T11:04:38.808 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 1 2023-10-11T11:04:39.807 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 info mirror/image_15@snap 2023-10-11T11:04:39.852 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:rbd image 'image_15': 2023-10-11T11:04:39.852 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: size 128 MiB in 32 objects 2023-10-11T11:04:39.852 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: order 22 (4 MiB objects) 2023-10-11T11:04:39.852 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: snapshot_count: 2 2023-10-11T11:04:39.853 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: id: 123d6205ec72 2023-10-11T11:04:39.853 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: block_name_prefix: rbd_data.123d6205ec72 2023-10-11T11:04:39.853 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: format: 2 2023-10-11T11:04:39.853 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-10-11T11:04:39.853 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: op_features: 2023-10-11T11:04:39.854 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: flags: 2023-10-11T11:04:39.854 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: create_timestamp: Wed Oct 11 10:56:29 2023 2023-10-11T11:04:39.854 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: access_timestamp: Wed Oct 11 10:56:29 2023 2023-10-11T11:04:39.854 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: modify_timestamp: Wed Oct 11 10:56:29 2023 2023-10-11T11:04:39.854 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: protected: False 2023-10-11T11:04:39.854 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring state: enabled 2023-10-11T11:04:39.855 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring mode: snapshot 2023-10-11T11:04:39.855 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring global id: 7f696646-7f90-4d09-a66b-990e8fbb123e 2023-10-11T11:04:39.855 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring primary: false 2023-10-11T11:04:39.856 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:04:39.856 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ compare_image_snaps mirror image_15 snap 2023-10-11T11:04:39.856 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:04:39.856 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_15 2023-10-11T11:04:39.857 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap 2023-10-11T11:04:39.857 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local ret=0 2023-10-11T11:04:39.857 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local rmt_export=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_15.export 2023-10-11T11:04:39.857 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local loc_export=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_15.export 2023-10-11T11:04:39.857 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rm -f /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_15.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_15.export 2023-10-11T11:04:39.858 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 -p mirror export image_15@snap /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_15.export 2023-10-11T11:04:40.036 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-10-11T11:04:40.043 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 -p mirror export image_15@snap /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_15.export 2023-10-11T11:04:40.290 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-10-11T11:04:40.296 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ cmp /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_15.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_15.export 2023-10-11T11:04:40.350 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rm -f /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_15.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_15.export 2023-10-11T11:04:40.388 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:04:40.388 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T11:04:40.388 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_16 2023-10-11T11:04:40.389 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ create_snap cluster2 mirror image_16 snap 2023-10-11T11:04:40.389 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:04:40.389 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:04:40.389 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_16 2023-10-11T11:04:40.389 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap 2023-10-11T11:04:40.390 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 -p mirror snap create image_16@snap --debug-rbd=20 --debug-journaler=20 2023-10-11T11:04:42.405 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_replay_started cluster1 mirror image_16 2023-10-11T11:04:42.406 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:04:42.406 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:04:42.406 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_16 2023-10-11T11:04:42.406 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_replay_state cluster1 mirror image_16 started 2023-10-11T11:04:42.406 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:04:42.407 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:04:42.407 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_16 2023-10-11T11:04:42.407 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local state=started 2023-10-11T11:04:42.407 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T11:04:42.407 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-10-11T11:04:42.408 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 1 2023-10-11T11:04:43.406 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test_image_replay_state cluster1 mirror image_16 started 2023-10-11T11:04:43.407 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:04:43.407 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:04:43.407 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_16 2023-10-11T11:04:43.407 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local test_state=started 2023-10-11T11:04:43.408 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local status_result 2023-10-11T11:04:43.408 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_state=stopped 2023-10-11T11:04:43.409 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_16 2023-10-11T11:04:43.409 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_instance=cluster1 2023-10-11T11:04:43.409 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T11:04:43.410 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ grep -i state 2023-10-11T11:04:43.410 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:04:43.410 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance=cluster1 2023-10-11T11:04:43.410 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local loop_instance 2023-10-11T11:04:43.410 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-10-11T11:04:43.411 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sleep 0 2023-10-11T11:04:43.411 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-10-11T11:04:43.411 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ seq 0 3 2023-10-11T11:04:43.413 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-10-11T11:04:43.413 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_16 2023-10-11T11:04:43.413 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1:0 2023-10-11T11:04:43.413 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T11:04:43.414 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance 2023-10-11T11:04:43.414 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-10-11T11:04:43.414 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local val=cluster1:0 2023-10-11T11:04:43.414 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_var_name=cluster 2023-10-11T11:04:43.414 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance_var_name=instance 2023-10-11T11:04:43.415 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ cluster=cluster1 2023-10-11T11:04:43.415 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ instance=0 2023-10-11T11:04:43.415 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' 0 = cluster1:0 ']' 2023-10-11T11:04:43.415 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval cluster=cluster1 2023-10-11T11:04:43.415 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:04:43.416 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval instance=0 2023-10-11T11:04:43.416 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=0 2023-10-11T11:04:43.416 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-10-11T11:04:43.416 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local local_cluster=cluster1:0 2023-10-11T11:04:43.418 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster=cluster1 2023-10-11T11:04:43.418 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance 2023-10-11T11:04:43.418 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-10-11T11:04:43.418 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local val=cluster1:0 2023-10-11T11:04:43.418 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster_var_name=local_cluster 2023-10-11T11:04:43.419 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance_var_name=instance 2023-10-11T11:04:43.419 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:04:43.419 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=0 2023-10-11T11:04:43.419 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ '[' 0 = cluster1:0 ']' 2023-10-11T11:04:43.419 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval local_cluster=cluster1 2023-10-11T11:04:43.420 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ local_cluster=cluster1 2023-10-11T11:04:43.420 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval instance=0 2023-10-11T11:04:43.420 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ instance=0 2023-10-11T11:04:43.420 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-10-11T11:04:43.433 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T11:04:43.433 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T11:04:43.434 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T11:04:43.434 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_16 2023-10-11T11:04:43.534 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:Invalid command: unused arguments: ['mirror/image_16'] 2023-10-11T11:04:43.534 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd mirror status : get status for rbd mirror 2023-10-11T11:04:43.535 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:admin_socket: invalid command 2023-10-11T11:04:43.537 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-10-11T11:04:43.537 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_16 2023-10-11T11:04:43.538 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1:1 2023-10-11T11:04:43.538 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T11:04:43.538 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance 2023-10-11T11:04:43.538 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-10-11T11:04:43.538 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local val=cluster1:1 2023-10-11T11:04:43.539 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_var_name=cluster 2023-10-11T11:04:43.539 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance_var_name=instance 2023-10-11T11:04:43.539 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ cluster=cluster1 2023-10-11T11:04:43.539 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ instance=1 2023-10-11T11:04:43.539 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' 1 = cluster1:1 ']' 2023-10-11T11:04:43.540 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval cluster=cluster1 2023-10-11T11:04:43.540 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:04:43.540 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval instance=1 2023-10-11T11:04:43.540 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=1 2023-10-11T11:04:43.541 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-10-11T11:04:43.541 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local local_cluster=cluster1:1 2023-10-11T11:04:43.541 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster=cluster1 2023-10-11T11:04:43.541 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance 2023-10-11T11:04:43.541 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-10-11T11:04:43.541 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local val=cluster1:1 2023-10-11T11:04:43.542 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster_var_name=local_cluster 2023-10-11T11:04:43.542 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance_var_name=instance 2023-10-11T11:04:43.542 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:04:43.542 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=1 2023-10-11T11:04:43.542 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ '[' 1 = cluster1:1 ']' 2023-10-11T11:04:43.543 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval local_cluster=cluster1 2023-10-11T11:04:43.543 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ local_cluster=cluster1 2023-10-11T11:04:43.543 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval instance=1 2023-10-11T11:04:43.543 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ instance=1 2023-10-11T11:04:43.543 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-10-11T11:04:43.556 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T11:04:43.556 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T11:04:43.556 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T11:04:43.557 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_16 2023-10-11T11:04:43.650 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:Invalid command: unused arguments: ['mirror/image_16'] 2023-10-11T11:04:43.650 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd mirror status : get status for rbd mirror 2023-10-11T11:04:43.650 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:admin_socket: invalid command 2023-10-11T11:04:43.653 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-10-11T11:04:43.653 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_16 2023-10-11T11:04:43.653 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1:2 2023-10-11T11:04:43.653 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T11:04:43.653 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance 2023-10-11T11:04:43.654 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-10-11T11:04:43.654 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local val=cluster1:2 2023-10-11T11:04:43.654 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_var_name=cluster 2023-10-11T11:04:43.654 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance_var_name=instance 2023-10-11T11:04:43.654 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ cluster=cluster1 2023-10-11T11:04:43.655 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ instance=2 2023-10-11T11:04:43.655 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' 2 = cluster1:2 ']' 2023-10-11T11:04:43.655 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval cluster=cluster1 2023-10-11T11:04:43.655 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:04:43.655 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval instance=2 2023-10-11T11:04:43.655 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=2 2023-10-11T11:04:43.656 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-10-11T11:04:43.656 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local local_cluster=cluster1:2 2023-10-11T11:04:43.656 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster=cluster1 2023-10-11T11:04:43.656 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance 2023-10-11T11:04:43.656 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-10-11T11:04:43.657 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local val=cluster1:2 2023-10-11T11:04:43.657 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster_var_name=local_cluster 2023-10-11T11:04:43.657 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance_var_name=instance 2023-10-11T11:04:43.657 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:04:43.657 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=2 2023-10-11T11:04:43.657 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ '[' 2 = cluster1:2 ']' 2023-10-11T11:04:43.658 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval local_cluster=cluster1 2023-10-11T11:04:43.658 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ local_cluster=cluster1 2023-10-11T11:04:43.658 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval instance=2 2023-10-11T11:04:43.658 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ instance=2 2023-10-11T11:04:43.658 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-10-11T11:04:43.671 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-10-11T11:04:43.671 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-10-11T11:04:43.671 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-10-11T11:04:43.672 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_16 2023-10-11T11:04:43.818 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ return 0 2023-10-11T11:04:43.819 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ status_result=' "state": "Replaying"' 2023-10-11T11:04:43.819 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ echo ' "state": "Replaying"' 2023-10-11T11:04:43.819 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ grep -i Replaying 2023-10-11T11:04:43.819 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: "state": "Replaying" 2023-10-11T11:04:43.821 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ current_state=started 2023-10-11T11:04:43.821 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test started = started 2023-10-11T11:04:43.822 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:04:43.822 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_16 2023-10-11T11:04:43.822 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local local_cluster=cluster1 2023-10-11T11:04:43.822 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:04:43.822 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:04:43.823 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_16 2023-10-11T11:04:43.823 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' snapshot = journal ']' 2023-10-11T11:04:43.823 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' snapshot = snapshot ']' 2023-10-11T11:04:43.823 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_16 2023-10-11T11:04:43.823 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local local_cluster=cluster1 2023-10-11T11:04:43.823 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:04:43.824 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:04:43.824 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_16 2023-10-11T11:04:43.824 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ mkfname cluster2-mirror-image_16.status 2023-10-11T11:04:43.824 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ echo cluster2-mirror-image_16.status 2023-10-11T11:04:43.824 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -e 's|[/ ]|_|g' 2023-10-11T11:04:43.825 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local status_log=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_16.status 2023-10-11T11:04:43.826 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ mkfname cluster1-mirror-image_16.status 2023-10-11T11:04:43.826 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ echo cluster1-mirror-image_16.status 2023-10-11T11:04:43.826 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -e 's|[/ ]|_|g' 2023-10-11T11:04:43.826 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local local_status_log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_16.status 2023-10-11T11:04:43.827 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ mirror_image_snapshot cluster2 mirror image_16 2023-10-11T11:04:43.827 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:04:43.827 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:04:43.827 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_16 2023-10-11T11:04:43.827 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_16 2023-10-11T11:04:44.407 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:Snapshot ID: 149 2023-10-11T11:04:44.418 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_16 /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_16.status 2023-10-11T11:04:44.418 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:04:44.418 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:04:44.419 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_16 2023-10-11T11:04:44.419 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_16.status 2023-10-11T11:04:44.419 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_16 --format xml 2023-10-11T11:04:44.419 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:04:44.461 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-10-11T11:04:44.463 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snapshot_id=149 2023-10-11T11:04:44.463 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T11:04:44.464 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T11:04:44.464 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.2 2023-10-11T11:04:44.584 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.3694610360410826) > weight (0.25) 2023-10-11T11:04:44.584 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.3311986999871259) > weight (0.25) 2023-10-11T11:04:44.585 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.2 2023-10-11T11:04:44.585 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Sent signal 15 2023-10-11T11:04:44.585 INFO:tasks.rbd_mirror_thrash:waiting for 1 secs before reviving daemons 2023-10-11T11:04:44.586 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi111.stderr:2023-10-11T11:04:44.585+0000 7fa39b03d700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 (PID: 116754) UID: 1000 2023-10-11T11:04:44.665 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_16 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_16.status 2023-10-11T11:04:44.666 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:04:44.666 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:04:44.666 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_16 2023-10-11T11:04:44.666 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_16.status 2023-10-11T11:04:44.666 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_16 --format xml 2023-10-11T11:04:44.668 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:04:44.726 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T11:04:44.727 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T11:04:44.728 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:-:1.1: Document is empty 2023-10-11T11:04:44.729 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: 2023-10-11T11:04:44.729 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:^ 2023-10-11T11:04:44.729 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id= 2023-10-11T11:04:44.729 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 149 = '' 2023-10-11T11:04:44.729 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T11:04:44.730 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.4 2023-10-11T11:04:45.130 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_16 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_16.status 2023-10-11T11:04:45.130 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:04:45.130 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:04:45.130 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_16 2023-10-11T11:04:45.131 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_16.status 2023-10-11T11:04:45.131 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_16 --format xml 2023-10-11T11:04:45.131 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:04:45.177 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T11:04:45.177 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T11:04:45.178 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:-:1.1: Document is empty 2023-10-11T11:04:45.179 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: 2023-10-11T11:04:45.179 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:^ 2023-10-11T11:04:45.179 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id= 2023-10-11T11:04:45.180 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 149 = '' 2023-10-11T11:04:45.180 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T11:04:45.180 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.8 2023-10-11T11:04:45.586 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.2 2023-10-11T11:04:45.586 INFO:teuthology.orchestra.run:waiting for 600 2023-10-11T11:04:45.593 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.2 2023-10-11T11:04:45.593 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Restarting daemon 2023-10-11T11:04:45.594 DEBUG:teuthology.orchestra.run.smithi111:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 2023-10-11T11:04:45.635 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Started 2023-10-11T11:04:45.636 INFO:tasks.rbd_mirror_thrash:waiting for 68 secs before thrashing 2023-10-11T11:04:45.980 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_16 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_16.status 2023-10-11T11:04:45.981 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:04:45.981 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:04:45.981 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_16 2023-10-11T11:04:45.981 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_16.status 2023-10-11T11:04:45.981 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_16 --format xml 2023-10-11T11:04:45.982 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:04:46.027 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T11:04:46.027 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T11:04:46.028 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:-:1.1: Document is empty 2023-10-11T11:04:46.028 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: 2023-10-11T11:04:46.029 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:^ 2023-10-11T11:04:46.029 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id= 2023-10-11T11:04:46.029 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 149 = '' 2023-10-11T11:04:46.030 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T11:04:46.030 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 1.6 2023-10-11T11:04:47.630 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_16 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_16.status 2023-10-11T11:04:47.631 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:04:47.631 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:04:47.631 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_16 2023-10-11T11:04:47.631 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_16.status 2023-10-11T11:04:47.631 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_16 --format xml 2023-10-11T11:04:47.632 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:04:47.682 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T11:04:47.682 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T11:04:47.683 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:-:1.1: Document is empty 2023-10-11T11:04:47.683 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: 2023-10-11T11:04:47.684 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:^ 2023-10-11T11:04:47.684 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id= 2023-10-11T11:04:47.685 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 149 = '' 2023-10-11T11:04:47.685 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T11:04:47.685 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 2 2023-10-11T11:04:49.685 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_16 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_16.status 2023-10-11T11:04:49.686 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:04:49.686 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:04:49.686 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_16 2023-10-11T11:04:49.686 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_16.status 2023-10-11T11:04:49.690 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:04:49.690 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_16 --format xml 2023-10-11T11:04:49.841 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T11:04:49.842 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T11:04:49.843 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:-:1.1: Document is empty 2023-10-11T11:04:49.843 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: 2023-10-11T11:04:49.843 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:^ 2023-10-11T11:04:49.844 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id= 2023-10-11T11:04:49.844 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 149 = '' 2023-10-11T11:04:49.844 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T11:04:49.844 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 2 2023-10-11T11:04:49.857 INFO:tasks.ceph.cluster1.osd.0.smithi050.stderr:2023-10-11T11:04:49.853+0000 7f61c5b4d700 -1 reset not still connected to 0x55651de5eb60 2023-10-11T11:04:49.857 INFO:tasks.ceph.cluster1.osd.0.smithi050.stderr:2023-10-11T11:04:49.853+0000 7f61c5b4d700 -1 reset not still connected to 0x55651de5ed00 2023-10-11T11:04:49.857 INFO:tasks.ceph.cluster1.osd.0.smithi050.stderr:2023-10-11T11:04:49.853+0000 7f61c5b4d700 -1 reset not still connected to 0x55651e5fe270 2023-10-11T11:04:49.858 INFO:tasks.ceph.cluster1.osd.0.smithi050.stderr:2023-10-11T11:04:49.853+0000 7f61c5b4d700 -1 reset not still connected to 0x55651e5fe820 2023-10-11T11:04:49.858 INFO:tasks.ceph.cluster1.osd.0.smithi050.stderr:2023-10-11T11:04:49.856+0000 7f61c5b4d700 -1 reset not still connected to 0x55651e5fed00 2023-10-11T11:04:49.859 INFO:tasks.ceph.cluster1.osd.0.smithi050.stderr:2023-10-11T11:04:49.856+0000 7f61c5b4d700 -1 reset not still connected to 0x55651e5feea0 2023-10-11T11:04:49.859 INFO:tasks.ceph.cluster1.osd.0.smithi050.stderr:2023-10-11T11:04:49.856+0000 7f61c5b4d700 -1 reset not still connected to 0x55651e5fef70 2023-10-11T11:04:49.860 INFO:tasks.ceph.cluster1.osd.0.smithi050.stderr:2023-10-11T11:04:49.857+0000 7f61c5b4d700 -1 reset not still connected to 0x55651e5ff380 2023-10-11T11:04:49.861 INFO:tasks.ceph.cluster1.osd.0.smithi050.stderr:2023-10-11T11:04:49.857+0000 7f61c5b4d700 -1 reset not still connected to 0x55651ecf1ee0 2023-10-11T11:04:49.907 INFO:tasks.ceph.cluster1.osd.0.smithi050.stderr:2023-10-11T11:04:49.903+0000 7f61c5b4d700 -1 reset not still connected to 0x55651b9b0dd0 2023-10-11T11:04:49.907 INFO:tasks.ceph.cluster1.osd.0.smithi050.stderr:2023-10-11T11:04:49.903+0000 7f61c5b4d700 -1 reset not still connected to 0x55651be28dd0 2023-10-11T11:04:49.907 INFO:tasks.ceph.cluster1.osd.0.smithi050.stderr:2023-10-11T11:04:49.903+0000 7f61c5b4d700 -1 reset not still connected to 0x55651ce38340 2023-10-11T11:04:49.907 INFO:tasks.ceph.cluster1.osd.0.smithi050.stderr:2023-10-11T11:04:49.903+0000 7f61c5b4d700 -1 reset not still connected to 0x55651d2ca1a0 2023-10-11T11:04:49.907 INFO:tasks.ceph.cluster1.osd.0.smithi050.stderr:2023-10-11T11:04:49.904+0000 7f61c5b4d700 -1 reset not still connected to 0x55651dbe04e0 2023-10-11T11:04:49.908 INFO:tasks.ceph.cluster1.osd.0.smithi050.stderr:2023-10-11T11:04:49.904+0000 7f61c5b4d700 -1 reset not still connected to 0x55651de5eb60 2023-10-11T11:04:49.908 INFO:tasks.ceph.cluster1.osd.0.smithi050.stderr:2023-10-11T11:04:49.904+0000 7f61c5b4d700 -1 reset not still connected to 0x55651de5ed00 2023-10-11T11:04:49.908 INFO:tasks.ceph.cluster1.osd.0.smithi050.stderr:2023-10-11T11:04:49.904+0000 7f61c5b4d700 -1 reset not still connected to 0x55651e5fe270 2023-10-11T11:04:49.908 INFO:tasks.ceph.cluster1.osd.0.smithi050.stderr:2023-10-11T11:04:49.904+0000 7f61c5b4d700 -1 reset not still connected to 0x55651e5fe680 2023-10-11T11:04:49.908 INFO:tasks.ceph.cluster1.osd.0.smithi050.stderr:2023-10-11T11:04:49.904+0000 7f61c5b4d700 -1 reset not still connected to 0x55651e5fe820 2023-10-11T11:04:49.908 INFO:tasks.ceph.cluster1.osd.0.smithi050.stderr:2023-10-11T11:04:49.904+0000 7f61c5b4d700 -1 reset not still connected to 0x55651e5fed00 2023-10-11T11:04:49.909 INFO:tasks.ceph.cluster1.osd.0.smithi050.stderr:2023-10-11T11:04:49.904+0000 7f61c5b4d700 -1 reset not still connected to 0x55651e5feea0 2023-10-11T11:04:49.909 INFO:tasks.ceph.cluster1.osd.0.smithi050.stderr:2023-10-11T11:04:49.904+0000 7f61c5b4d700 -1 reset not still connected to 0x55651ececea0 2023-10-11T11:04:49.909 INFO:tasks.ceph.cluster1.osd.0.smithi050.stderr:2023-10-11T11:04:49.904+0000 7f61c5b4d700 -1 reset not still connected to 0x55651ecf1ee0 2023-10-11T11:04:51.845 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_16 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_16.status 2023-10-11T11:04:51.845 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:04:51.845 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:04:51.846 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_16 2023-10-11T11:04:51.846 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_16.status 2023-10-11T11:04:51.846 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_16 --format xml 2023-10-11T11:04:51.846 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:04:51.898 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T11:04:51.899 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T11:04:51.900 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:-:1.1: Document is empty 2023-10-11T11:04:51.900 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: 2023-10-11T11:04:51.900 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:^ 2023-10-11T11:04:51.901 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id= 2023-10-11T11:04:51.901 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 149 = '' 2023-10-11T11:04:51.902 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T11:04:51.902 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 4 2023-10-11T11:04:55.903 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_16 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_16.status 2023-10-11T11:04:55.903 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:04:55.903 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:04:55.904 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_16 2023-10-11T11:04:55.904 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_16.status 2023-10-11T11:04:55.904 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_16 --format xml 2023-10-11T11:04:55.904 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:04:55.955 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T11:04:55.957 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id=149 2023-10-11T11:04:55.957 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 149 = 149 2023-10-11T11:04:55.958 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:04:55.958 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_snap_present cluster1 mirror image_16 snap 2023-10-11T11:04:55.958 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:04:55.958 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:04:55.959 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_16 2023-10-11T11:04:55.959 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap 2023-10-11T11:04:55.959 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T11:04:55.959 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-10-11T11:04:55.960 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 1 2023-10-11T11:04:56.958 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 info mirror/image_16@snap 2023-10-11T11:04:57.005 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:rbd image 'image_16': 2023-10-11T11:04:57.005 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: size 128 MiB in 32 objects 2023-10-11T11:04:57.005 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: order 22 (4 MiB objects) 2023-10-11T11:04:57.005 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: snapshot_count: 2 2023-10-11T11:04:57.006 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: id: 123daca2aaa7 2023-10-11T11:04:57.006 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: block_name_prefix: rbd_data.123daca2aaa7 2023-10-11T11:04:57.006 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: format: 2 2023-10-11T11:04:57.006 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-10-11T11:04:57.006 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: op_features: 2023-10-11T11:04:57.006 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: flags: 2023-10-11T11:04:57.007 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: create_timestamp: Wed Oct 11 10:56:44 2023 2023-10-11T11:04:57.007 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: access_timestamp: Wed Oct 11 10:56:44 2023 2023-10-11T11:04:57.007 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: modify_timestamp: Wed Oct 11 10:56:44 2023 2023-10-11T11:04:57.007 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: protected: False 2023-10-11T11:04:57.007 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring state: enabled 2023-10-11T11:04:57.008 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring mode: snapshot 2023-10-11T11:04:57.008 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring global id: 2ba7e4f0-06a3-4db0-aeee-64ab913d9bf0 2023-10-11T11:04:57.008 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring primary: false 2023-10-11T11:04:57.009 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:04:57.009 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ compare_image_snaps mirror image_16 snap 2023-10-11T11:04:57.009 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:04:57.009 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_16 2023-10-11T11:04:57.010 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap 2023-10-11T11:04:57.010 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local ret=0 2023-10-11T11:04:57.010 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local rmt_export=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_16.export 2023-10-11T11:04:57.010 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local loc_export=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_16.export 2023-10-11T11:04:57.010 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rm -f /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_16.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_16.export 2023-10-11T11:04:57.011 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 -p mirror export image_16@snap /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_16.export 2023-10-11T11:04:57.187 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-10-11T11:04:57.194 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 -p mirror export image_16@snap /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_16.export 2023-10-11T11:04:57.418 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-10-11T11:04:57.424 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ cmp /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_16.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_16.export 2023-10-11T11:04:57.478 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rm -f /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_16.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_16.export 2023-10-11T11:04:57.515 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:04:57.516 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T11:04:57.516 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_17 2023-10-11T11:04:57.516 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ create_snap cluster2 mirror image_17 snap 2023-10-11T11:04:57.516 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:04:57.516 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:04:57.517 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_17 2023-10-11T11:04:57.517 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap 2023-10-11T11:04:57.517 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 -p mirror snap create image_17@snap --debug-rbd=20 --debug-journaler=20 2023-10-11T11:04:59.496 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_replay_started cluster1 mirror image_17 2023-10-11T11:04:59.496 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:04:59.496 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:04:59.496 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_17 2023-10-11T11:04:59.497 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_replay_state cluster1 mirror image_17 started 2023-10-11T11:04:59.497 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:04:59.497 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:04:59.497 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_17 2023-10-11T11:04:59.497 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local state=started 2023-10-11T11:04:59.497 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T11:04:59.498 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-10-11T11:04:59.498 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 1 2023-10-11T11:05:00.498 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test_image_replay_state cluster1 mirror image_17 started 2023-10-11T11:05:00.499 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:05:00.499 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:05:00.499 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_17 2023-10-11T11:05:00.500 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local test_state=started 2023-10-11T11:05:00.500 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local status_result 2023-10-11T11:05:00.500 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_state=stopped 2023-10-11T11:05:00.500 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_17 2023-10-11T11:05:00.501 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_instance=cluster1 2023-10-11T11:05:00.501 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T11:05:00.501 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ grep -i state 2023-10-11T11:05:00.501 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:05:00.502 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance=cluster1 2023-10-11T11:05:00.502 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local loop_instance 2023-10-11T11:05:00.502 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-10-11T11:05:00.502 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sleep 0 2023-10-11T11:05:00.506 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-10-11T11:05:00.506 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ seq 0 3 2023-10-11T11:05:00.506 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-10-11T11:05:00.507 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_17 2023-10-11T11:05:00.507 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1:0 2023-10-11T11:05:00.507 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T11:05:00.507 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance 2023-10-11T11:05:00.508 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-10-11T11:05:00.508 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local val=cluster1:0 2023-10-11T11:05:00.508 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_var_name=cluster 2023-10-11T11:05:00.508 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance_var_name=instance 2023-10-11T11:05:00.509 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ cluster=cluster1 2023-10-11T11:05:00.509 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ instance=0 2023-10-11T11:05:00.509 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' 0 = cluster1:0 ']' 2023-10-11T11:05:00.509 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval cluster=cluster1 2023-10-11T11:05:00.510 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:05:00.510 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval instance=0 2023-10-11T11:05:00.510 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=0 2023-10-11T11:05:00.510 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-10-11T11:05:00.511 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local local_cluster=cluster1:0 2023-10-11T11:05:00.511 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster=cluster1 2023-10-11T11:05:00.511 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance 2023-10-11T11:05:00.511 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-10-11T11:05:00.512 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local val=cluster1:0 2023-10-11T11:05:00.512 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster_var_name=local_cluster 2023-10-11T11:05:00.512 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance_var_name=instance 2023-10-11T11:05:00.512 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:05:00.513 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=0 2023-10-11T11:05:00.513 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ '[' 0 = cluster1:0 ']' 2023-10-11T11:05:00.513 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval local_cluster=cluster1 2023-10-11T11:05:00.513 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ local_cluster=cluster1 2023-10-11T11:05:00.513 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval instance=0 2023-10-11T11:05:00.514 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ instance=0 2023-10-11T11:05:00.514 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-10-11T11:05:00.524 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T11:05:00.524 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T11:05:00.525 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T11:05:00.525 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_17 2023-10-11T11:05:00.625 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:Invalid command: unused arguments: ['mirror/image_17'] 2023-10-11T11:05:00.626 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd mirror status : get status for rbd mirror 2023-10-11T11:05:00.626 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:admin_socket: invalid command 2023-10-11T11:05:00.629 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-10-11T11:05:00.630 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_17 2023-10-11T11:05:00.630 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1:1 2023-10-11T11:05:00.630 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T11:05:00.630 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance 2023-10-11T11:05:00.631 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-10-11T11:05:00.631 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local val=cluster1:1 2023-10-11T11:05:00.631 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_var_name=cluster 2023-10-11T11:05:00.631 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance_var_name=instance 2023-10-11T11:05:00.632 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ cluster=cluster1 2023-10-11T11:05:00.632 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ instance=1 2023-10-11T11:05:00.632 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' 1 = cluster1:1 ']' 2023-10-11T11:05:00.632 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval cluster=cluster1 2023-10-11T11:05:00.633 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:05:00.633 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval instance=1 2023-10-11T11:05:00.633 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=1 2023-10-11T11:05:00.633 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-10-11T11:05:00.634 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local local_cluster=cluster1:1 2023-10-11T11:05:00.634 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster=cluster1 2023-10-11T11:05:00.634 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance 2023-10-11T11:05:00.634 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-10-11T11:05:00.635 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local val=cluster1:1 2023-10-11T11:05:00.635 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster_var_name=local_cluster 2023-10-11T11:05:00.635 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance_var_name=instance 2023-10-11T11:05:00.635 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:05:00.636 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=1 2023-10-11T11:05:00.636 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ '[' 1 = cluster1:1 ']' 2023-10-11T11:05:00.636 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval local_cluster=cluster1 2023-10-11T11:05:00.636 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ local_cluster=cluster1 2023-10-11T11:05:00.637 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval instance=1 2023-10-11T11:05:00.637 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ instance=1 2023-10-11T11:05:00.637 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-10-11T11:05:00.648 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T11:05:00.648 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T11:05:00.648 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T11:05:00.648 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_17 2023-10-11T11:05:00.797 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ return 0 2023-10-11T11:05:00.797 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ status_result=' "state": "Replaying"' 2023-10-11T11:05:00.797 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ echo ' "state": "Replaying"' 2023-10-11T11:05:00.798 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ grep -i Replaying 2023-10-11T11:05:00.798 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: "state": "Replaying" 2023-10-11T11:05:00.801 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ current_state=started 2023-10-11T11:05:00.801 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test started = started 2023-10-11T11:05:00.801 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:05:00.802 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_17 2023-10-11T11:05:00.802 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local local_cluster=cluster1 2023-10-11T11:05:00.802 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:05:00.803 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:05:00.803 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_17 2023-10-11T11:05:00.803 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' snapshot = journal ']' 2023-10-11T11:05:00.803 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' snapshot = snapshot ']' 2023-10-11T11:05:00.804 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_17 2023-10-11T11:05:00.804 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local local_cluster=cluster1 2023-10-11T11:05:00.804 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:05:00.804 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:05:00.805 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_17 2023-10-11T11:05:00.805 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ mkfname cluster2-mirror-image_17.status 2023-10-11T11:05:00.805 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ echo cluster2-mirror-image_17.status 2023-10-11T11:05:00.805 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -e 's|[/ ]|_|g' 2023-10-11T11:05:00.806 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local status_log=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_17.status 2023-10-11T11:05:00.806 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ mkfname cluster1-mirror-image_17.status 2023-10-11T11:05:00.806 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ echo cluster1-mirror-image_17.status 2023-10-11T11:05:00.806 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -e 's|[/ ]|_|g' 2023-10-11T11:05:00.807 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local local_status_log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_17.status 2023-10-11T11:05:00.808 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ mirror_image_snapshot cluster2 mirror image_17 2023-10-11T11:05:00.808 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:05:00.808 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:05:00.808 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_17 2023-10-11T11:05:00.809 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_17 2023-10-11T11:05:01.494 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:Snapshot ID: 151 2023-10-11T11:05:01.505 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_17 /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_17.status 2023-10-11T11:05:01.505 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:05:01.505 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:05:01.506 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_17 2023-10-11T11:05:01.506 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_17.status 2023-10-11T11:05:01.506 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_17 --format xml 2023-10-11T11:05:01.506 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:05:01.548 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-10-11T11:05:01.550 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snapshot_id=151 2023-10-11T11:05:01.550 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T11:05:01.550 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T11:05:01.551 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.2 2023-10-11T11:05:01.751 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_17 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_17.status 2023-10-11T11:05:01.751 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:05:01.752 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:05:01.752 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_17 2023-10-11T11:05:01.752 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_17.status 2023-10-11T11:05:01.752 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_17 --format xml 2023-10-11T11:05:01.752 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:05:01.797 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T11:05:01.798 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T11:05:01.799 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:-:1.1: Document is empty 2023-10-11T11:05:01.799 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: 2023-10-11T11:05:01.799 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:^ 2023-10-11T11:05:01.800 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id= 2023-10-11T11:05:01.800 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 151 = '' 2023-10-11T11:05:01.800 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T11:05:01.800 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.4 2023-10-11T11:05:02.201 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_17 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_17.status 2023-10-11T11:05:02.201 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:05:02.201 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:05:02.201 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_17 2023-10-11T11:05:02.201 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_17.status 2023-10-11T11:05:02.202 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_17 --format xml 2023-10-11T11:05:02.202 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:05:02.249 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T11:05:02.249 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T11:05:02.250 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:-:1.1: Document is empty 2023-10-11T11:05:02.250 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: 2023-10-11T11:05:02.251 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:^ 2023-10-11T11:05:02.251 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id= 2023-10-11T11:05:02.252 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 151 = '' 2023-10-11T11:05:02.252 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T11:05:02.252 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.8 2023-10-11T11:05:03.052 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_17 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_17.status 2023-10-11T11:05:03.052 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:05:03.052 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:05:03.053 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_17 2023-10-11T11:05:03.053 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_17.status 2023-10-11T11:05:03.053 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_17 --format xml 2023-10-11T11:05:03.053 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:05:03.098 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T11:05:03.100 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id=151 2023-10-11T11:05:03.100 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 151 = 151 2023-10-11T11:05:03.101 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:05:03.101 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_snap_present cluster1 mirror image_17 snap 2023-10-11T11:05:03.101 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:05:03.101 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:05:03.101 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_17 2023-10-11T11:05:03.102 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap 2023-10-11T11:05:03.102 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T11:05:03.102 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-10-11T11:05:03.102 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 1 2023-10-11T11:05:04.101 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 info mirror/image_17@snap 2023-10-11T11:05:04.147 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:rbd image 'image_17': 2023-10-11T11:05:04.147 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: size 128 MiB in 32 objects 2023-10-11T11:05:04.147 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: order 22 (4 MiB objects) 2023-10-11T11:05:04.148 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: snapshot_count: 2 2023-10-11T11:05:04.148 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: id: 123dcc3acdef 2023-10-11T11:05:04.148 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: block_name_prefix: rbd_data.123dcc3acdef 2023-10-11T11:05:04.148 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: format: 2 2023-10-11T11:05:04.149 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-10-11T11:05:04.149 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: op_features: 2023-10-11T11:05:04.149 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: flags: 2023-10-11T11:05:04.149 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: create_timestamp: Wed Oct 11 10:56:53 2023 2023-10-11T11:05:04.150 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: access_timestamp: Wed Oct 11 10:56:53 2023 2023-10-11T11:05:04.150 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: modify_timestamp: Wed Oct 11 10:56:53 2023 2023-10-11T11:05:04.150 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: protected: False 2023-10-11T11:05:04.150 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring state: enabled 2023-10-11T11:05:04.151 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring mode: snapshot 2023-10-11T11:05:04.151 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring global id: f3a165bf-82fd-4110-9215-0cf728d304ac 2023-10-11T11:05:04.151 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring primary: false 2023-10-11T11:05:04.153 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:05:04.153 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ compare_image_snaps mirror image_17 snap 2023-10-11T11:05:04.153 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:05:04.154 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_17 2023-10-11T11:05:04.154 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap 2023-10-11T11:05:04.154 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local ret=0 2023-10-11T11:05:04.155 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local rmt_export=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_17.export 2023-10-11T11:05:04.155 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local loc_export=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_17.export 2023-10-11T11:05:04.155 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rm -f /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_17.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_17.export 2023-10-11T11:05:04.155 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 -p mirror export image_17@snap /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_17.export 2023-10-11T11:05:04.332 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-10-11T11:05:04.338 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 -p mirror export image_17@snap /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_17.export 2023-10-11T11:05:04.560 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-10-11T11:05:04.567 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ cmp /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_17.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_17.export 2023-10-11T11:05:04.621 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rm -f /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_17.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_17.export 2023-10-11T11:05:04.658 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:05:04.658 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T11:05:04.659 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_18 2023-10-11T11:05:04.659 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ create_snap cluster2 mirror image_18 snap 2023-10-11T11:05:04.659 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:05:04.659 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:05:04.660 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_18 2023-10-11T11:05:04.660 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap 2023-10-11T11:05:04.660 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 -p mirror snap create image_18@snap --debug-rbd=20 --debug-journaler=20 2023-10-11T11:05:06.508 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_replay_started cluster1 mirror image_18 2023-10-11T11:05:06.509 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:05:06.509 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:05:06.509 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_18 2023-10-11T11:05:06.509 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_replay_state cluster1 mirror image_18 started 2023-10-11T11:05:06.509 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:05:06.509 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:05:06.510 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_18 2023-10-11T11:05:06.510 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local state=started 2023-10-11T11:05:06.510 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T11:05:06.510 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-10-11T11:05:06.510 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 1 2023-10-11T11:05:07.510 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test_image_replay_state cluster1 mirror image_18 started 2023-10-11T11:05:07.511 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:05:07.511 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:05:07.511 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_18 2023-10-11T11:05:07.511 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local test_state=started 2023-10-11T11:05:07.511 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local status_result 2023-10-11T11:05:07.512 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_state=stopped 2023-10-11T11:05:07.512 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_18 2023-10-11T11:05:07.512 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_instance=cluster1 2023-10-11T11:05:07.512 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T11:05:07.512 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:05:07.513 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ grep -i state 2023-10-11T11:05:07.513 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance=cluster1 2023-10-11T11:05:07.513 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local loop_instance 2023-10-11T11:05:07.513 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-10-11T11:05:07.513 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sleep 0 2023-10-11T11:05:07.515 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-10-11T11:05:07.515 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ seq 0 3 2023-10-11T11:05:07.516 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-10-11T11:05:07.516 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_18 2023-10-11T11:05:07.516 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1:0 2023-10-11T11:05:07.516 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T11:05:07.516 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance 2023-10-11T11:05:07.516 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-10-11T11:05:07.517 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local val=cluster1:0 2023-10-11T11:05:07.517 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_var_name=cluster 2023-10-11T11:05:07.517 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance_var_name=instance 2023-10-11T11:05:07.517 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ cluster=cluster1 2023-10-11T11:05:07.517 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ instance=0 2023-10-11T11:05:07.518 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' 0 = cluster1:0 ']' 2023-10-11T11:05:07.518 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval cluster=cluster1 2023-10-11T11:05:07.518 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:05:07.518 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval instance=0 2023-10-11T11:05:07.518 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=0 2023-10-11T11:05:07.518 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-10-11T11:05:07.519 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local local_cluster=cluster1:0 2023-10-11T11:05:07.519 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster=cluster1 2023-10-11T11:05:07.519 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance 2023-10-11T11:05:07.519 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-10-11T11:05:07.519 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local val=cluster1:0 2023-10-11T11:05:07.519 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster_var_name=local_cluster 2023-10-11T11:05:07.520 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance_var_name=instance 2023-10-11T11:05:07.521 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:05:07.521 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=0 2023-10-11T11:05:07.521 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ '[' 0 = cluster1:0 ']' 2023-10-11T11:05:07.521 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval local_cluster=cluster1 2023-10-11T11:05:07.521 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ local_cluster=cluster1 2023-10-11T11:05:07.521 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval instance=0 2023-10-11T11:05:07.522 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ instance=0 2023-10-11T11:05:07.522 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-10-11T11:05:07.538 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T11:05:07.538 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T11:05:07.539 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T11:05:07.539 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_18 2023-10-11T11:05:07.639 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:Invalid command: unused arguments: ['mirror/image_18'] 2023-10-11T11:05:07.639 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd mirror status : get status for rbd mirror 2023-10-11T11:05:07.639 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:admin_socket: invalid command 2023-10-11T11:05:07.642 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-10-11T11:05:07.642 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_18 2023-10-11T11:05:07.642 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1:1 2023-10-11T11:05:07.642 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T11:05:07.643 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance 2023-10-11T11:05:07.643 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-10-11T11:05:07.643 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local val=cluster1:1 2023-10-11T11:05:07.643 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_var_name=cluster 2023-10-11T11:05:07.643 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance_var_name=instance 2023-10-11T11:05:07.644 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ cluster=cluster1 2023-10-11T11:05:07.644 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ instance=1 2023-10-11T11:05:07.644 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' 1 = cluster1:1 ']' 2023-10-11T11:05:07.644 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval cluster=cluster1 2023-10-11T11:05:07.644 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:05:07.644 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval instance=1 2023-10-11T11:05:07.644 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=1 2023-10-11T11:05:07.645 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-10-11T11:05:07.645 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local local_cluster=cluster1:1 2023-10-11T11:05:07.645 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster=cluster1 2023-10-11T11:05:07.645 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance 2023-10-11T11:05:07.645 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-10-11T11:05:07.646 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local val=cluster1:1 2023-10-11T11:05:07.646 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster_var_name=local_cluster 2023-10-11T11:05:07.646 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance_var_name=instance 2023-10-11T11:05:07.646 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:05:07.646 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=1 2023-10-11T11:05:07.646 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ '[' 1 = cluster1:1 ']' 2023-10-11T11:05:07.647 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval local_cluster=cluster1 2023-10-11T11:05:07.647 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ local_cluster=cluster1 2023-10-11T11:05:07.647 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval instance=1 2023-10-11T11:05:07.647 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ instance=1 2023-10-11T11:05:07.647 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-10-11T11:05:07.661 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T11:05:07.661 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T11:05:07.662 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T11:05:07.662 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_18 2023-10-11T11:05:07.809 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ return 0 2023-10-11T11:05:07.810 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ status_result=' "state": "Replaying"' 2023-10-11T11:05:07.810 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ echo ' "state": "Replaying"' 2023-10-11T11:05:07.810 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ grep -i Replaying 2023-10-11T11:05:07.811 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: "state": "Replaying" 2023-10-11T11:05:07.812 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ current_state=started 2023-10-11T11:05:07.812 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test started = started 2023-10-11T11:05:07.813 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:05:07.813 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_18 2023-10-11T11:05:07.813 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local local_cluster=cluster1 2023-10-11T11:05:07.813 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:05:07.813 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:05:07.813 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_18 2023-10-11T11:05:07.814 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' snapshot = journal ']' 2023-10-11T11:05:07.814 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' snapshot = snapshot ']' 2023-10-11T11:05:07.814 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_18 2023-10-11T11:05:07.814 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local local_cluster=cluster1 2023-10-11T11:05:07.814 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:05:07.814 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:05:07.815 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_18 2023-10-11T11:05:07.815 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ mkfname cluster2-mirror-image_18.status 2023-10-11T11:05:07.815 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ echo cluster2-mirror-image_18.status 2023-10-11T11:05:07.815 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -e 's|[/ ]|_|g' 2023-10-11T11:05:07.816 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local status_log=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_18.status 2023-10-11T11:05:07.817 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ mkfname cluster1-mirror-image_18.status 2023-10-11T11:05:07.817 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ echo cluster1-mirror-image_18.status 2023-10-11T11:05:07.817 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -e 's|[/ ]|_|g' 2023-10-11T11:05:07.817 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local local_status_log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_18.status 2023-10-11T11:05:07.817 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ mirror_image_snapshot cluster2 mirror image_18 2023-10-11T11:05:07.817 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:05:07.818 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:05:07.818 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_18 2023-10-11T11:05:07.818 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_18 2023-10-11T11:05:08.509 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:Snapshot ID: 153 2023-10-11T11:05:08.520 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_18 /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_18.status 2023-10-11T11:05:08.520 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:05:08.521 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:05:08.521 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_18 2023-10-11T11:05:08.521 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_18.status 2023-10-11T11:05:08.521 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_18 --format xml 2023-10-11T11:05:08.521 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:05:08.564 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-10-11T11:05:08.566 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snapshot_id=153 2023-10-11T11:05:08.566 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T11:05:08.566 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T11:05:08.567 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.2 2023-10-11T11:05:08.767 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_18 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_18.status 2023-10-11T11:05:08.767 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:05:08.767 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:05:08.768 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_18 2023-10-11T11:05:08.768 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_18.status 2023-10-11T11:05:08.768 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_18 --format xml 2023-10-11T11:05:08.768 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:05:08.815 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T11:05:08.815 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T11:05:08.816 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:-:1.1: Document is empty 2023-10-11T11:05:08.817 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: 2023-10-11T11:05:08.817 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:^ 2023-10-11T11:05:08.817 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id= 2023-10-11T11:05:08.818 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 153 = '' 2023-10-11T11:05:08.818 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T11:05:08.818 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.4 2023-10-11T11:05:09.218 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_18 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_18.status 2023-10-11T11:05:09.218 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:05:09.218 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:05:09.219 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_18 2023-10-11T11:05:09.219 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_18.status 2023-10-11T11:05:09.219 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_18 --format xml 2023-10-11T11:05:09.219 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:05:09.268 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T11:05:09.268 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T11:05:09.269 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:-:1.1: Document is empty 2023-10-11T11:05:09.269 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: 2023-10-11T11:05:09.270 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:^ 2023-10-11T11:05:09.270 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id= 2023-10-11T11:05:09.271 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 153 = '' 2023-10-11T11:05:09.271 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T11:05:09.271 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.8 2023-10-11T11:05:09.564 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2023-10-11T11:05:09.565 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2023-10-11T11:05:09.565 INFO:tasks.rbd_mirror_thrash:waiting for 4 secs before reviving daemons 2023-10-11T11:05:09.566 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi111.stderr:2023-10-11T11:05:09.565+0000 7f37dd0ea700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 117023) UID: 1000 2023-10-11T11:05:10.071 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_18 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_18.status 2023-10-11T11:05:10.072 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:05:10.072 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:05:10.072 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_18 2023-10-11T11:05:10.072 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_18.status 2023-10-11T11:05:10.072 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_18 --format xml 2023-10-11T11:05:10.073 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:05:10.115 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T11:05:10.117 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id=153 2023-10-11T11:05:10.117 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 153 = 153 2023-10-11T11:05:10.118 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:05:10.118 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_snap_present cluster1 mirror image_18 snap 2023-10-11T11:05:10.118 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:05:10.118 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:05:10.118 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_18 2023-10-11T11:05:10.118 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap 2023-10-11T11:05:10.119 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T11:05:10.119 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-10-11T11:05:10.119 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 1 2023-10-11T11:05:11.118 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 info mirror/image_18@snap 2023-10-11T11:05:11.159 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:rbd image 'image_18': 2023-10-11T11:05:11.160 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: size 128 MiB in 32 objects 2023-10-11T11:05:11.160 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: order 22 (4 MiB objects) 2023-10-11T11:05:11.160 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: snapshot_count: 2 2023-10-11T11:05:11.161 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: id: 123de8a1d5a5 2023-10-11T11:05:11.161 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: block_name_prefix: rbd_data.123de8a1d5a5 2023-10-11T11:05:11.161 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: format: 2 2023-10-11T11:05:11.161 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-10-11T11:05:11.162 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: op_features: 2023-10-11T11:05:11.162 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: flags: 2023-10-11T11:05:11.162 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: create_timestamp: Wed Oct 11 10:57:08 2023 2023-10-11T11:05:11.162 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: access_timestamp: Wed Oct 11 10:57:08 2023 2023-10-11T11:05:11.163 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: modify_timestamp: Wed Oct 11 10:57:08 2023 2023-10-11T11:05:11.163 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: protected: False 2023-10-11T11:05:11.163 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring state: enabled 2023-10-11T11:05:11.163 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring mode: snapshot 2023-10-11T11:05:11.164 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring global id: 1526990d-f216-41c2-accc-30ee745ab7bd 2023-10-11T11:05:11.164 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring primary: false 2023-10-11T11:05:11.165 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:05:11.165 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ compare_image_snaps mirror image_18 snap 2023-10-11T11:05:11.166 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:05:11.166 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_18 2023-10-11T11:05:11.166 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap 2023-10-11T11:05:11.167 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local ret=0 2023-10-11T11:05:11.167 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local rmt_export=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_18.export 2023-10-11T11:05:11.167 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local loc_export=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_18.export 2023-10-11T11:05:11.167 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rm -f /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_18.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_18.export 2023-10-11T11:05:11.168 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 -p mirror export image_18@snap /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_18.export 2023-10-11T11:05:11.346 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-10-11T11:05:11.352 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 -p mirror export image_18@snap /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_18.export 2023-10-11T11:05:11.568 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-10-11T11:05:11.575 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ cmp /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_18.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_18.export 2023-10-11T11:05:11.629 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rm -f /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_18.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_18.export 2023-10-11T11:05:11.666 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:05:11.666 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T11:05:11.666 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_19 2023-10-11T11:05:11.667 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ create_snap cluster2 mirror image_19 snap 2023-10-11T11:05:11.667 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:05:11.667 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:05:11.667 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_19 2023-10-11T11:05:11.667 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap 2023-10-11T11:05:11.667 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 -p mirror snap create image_19@snap --debug-rbd=20 --debug-journaler=20 2023-10-11T11:05:13.410 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_replay_started cluster1 mirror image_19 2023-10-11T11:05:13.410 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:05:13.410 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:05:13.410 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_19 2023-10-11T11:05:13.411 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_replay_state cluster1 mirror image_19 started 2023-10-11T11:05:13.411 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:05:13.411 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:05:13.411 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_19 2023-10-11T11:05:13.411 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local state=started 2023-10-11T11:05:13.412 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T11:05:13.412 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-10-11T11:05:13.412 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 1 2023-10-11T11:05:13.565 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2023-10-11T11:05:13.566 INFO:teuthology.orchestra.run:waiting for 600 2023-10-11T11:05:13.566 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2023-10-11T11:05:13.566 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2023-10-11T11:05:13.567 DEBUG:teuthology.orchestra.run.smithi111:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2023-10-11T11:05:13.570 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2023-10-11T11:05:13.570 INFO:tasks.rbd_mirror_thrash:waiting for 110 secs before thrashing 2023-10-11T11:05:14.412 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test_image_replay_state cluster1 mirror image_19 started 2023-10-11T11:05:14.412 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:05:14.413 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:05:14.413 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_19 2023-10-11T11:05:14.413 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local test_state=started 2023-10-11T11:05:14.414 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local status_result 2023-10-11T11:05:14.414 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_state=stopped 2023-10-11T11:05:14.414 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_19 2023-10-11T11:05:14.414 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ grep -i state 2023-10-11T11:05:14.415 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_instance=cluster1 2023-10-11T11:05:14.415 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T11:05:14.415 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:05:14.415 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance=cluster1 2023-10-11T11:05:14.416 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local loop_instance 2023-10-11T11:05:14.416 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-10-11T11:05:14.416 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sleep 0 2023-10-11T11:05:14.420 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-10-11T11:05:14.420 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ seq 0 3 2023-10-11T11:05:14.420 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-10-11T11:05:14.420 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_19 2023-10-11T11:05:14.421 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1:0 2023-10-11T11:05:14.421 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T11:05:14.421 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance 2023-10-11T11:05:14.421 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-10-11T11:05:14.421 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local val=cluster1:0 2023-10-11T11:05:14.422 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_var_name=cluster 2023-10-11T11:05:14.422 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance_var_name=instance 2023-10-11T11:05:14.422 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ cluster=cluster1 2023-10-11T11:05:14.423 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ instance=0 2023-10-11T11:05:14.423 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' 0 = cluster1:0 ']' 2023-10-11T11:05:14.423 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval cluster=cluster1 2023-10-11T11:05:14.423 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:05:14.423 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval instance=0 2023-10-11T11:05:14.424 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=0 2023-10-11T11:05:14.424 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-10-11T11:05:14.424 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local local_cluster=cluster1:0 2023-10-11T11:05:14.424 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster=cluster1 2023-10-11T11:05:14.425 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance 2023-10-11T11:05:14.425 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-10-11T11:05:14.425 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local val=cluster1:0 2023-10-11T11:05:14.425 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster_var_name=local_cluster 2023-10-11T11:05:14.425 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance_var_name=instance 2023-10-11T11:05:14.426 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:05:14.426 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=0 2023-10-11T11:05:14.426 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ '[' 0 = cluster1:0 ']' 2023-10-11T11:05:14.426 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval local_cluster=cluster1 2023-10-11T11:05:14.427 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ local_cluster=cluster1 2023-10-11T11:05:14.427 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval instance=0 2023-10-11T11:05:14.427 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ instance=0 2023-10-11T11:05:14.427 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-10-11T11:05:14.437 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T11:05:14.437 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T11:05:14.438 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T11:05:14.438 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_19 2023-10-11T11:05:14.535 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:Invalid command: unused arguments: ['mirror/image_19'] 2023-10-11T11:05:14.535 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd mirror status : get status for rbd mirror 2023-10-11T11:05:14.535 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:admin_socket: invalid command 2023-10-11T11:05:14.539 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-10-11T11:05:14.539 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_19 2023-10-11T11:05:14.540 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1:1 2023-10-11T11:05:14.540 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T11:05:14.540 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance 2023-10-11T11:05:14.540 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-10-11T11:05:14.541 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local val=cluster1:1 2023-10-11T11:05:14.541 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_var_name=cluster 2023-10-11T11:05:14.541 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance_var_name=instance 2023-10-11T11:05:14.541 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ cluster=cluster1 2023-10-11T11:05:14.541 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ instance=1 2023-10-11T11:05:14.542 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' 1 = cluster1:1 ']' 2023-10-11T11:05:14.542 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval cluster=cluster1 2023-10-11T11:05:14.542 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:05:14.543 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval instance=1 2023-10-11T11:05:14.543 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=1 2023-10-11T11:05:14.543 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-10-11T11:05:14.543 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local local_cluster=cluster1:1 2023-10-11T11:05:14.544 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster=cluster1 2023-10-11T11:05:14.544 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance 2023-10-11T11:05:14.544 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-10-11T11:05:14.544 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local val=cluster1:1 2023-10-11T11:05:14.545 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster_var_name=local_cluster 2023-10-11T11:05:14.545 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance_var_name=instance 2023-10-11T11:05:14.545 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:05:14.545 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=1 2023-10-11T11:05:14.545 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ '[' 1 = cluster1:1 ']' 2023-10-11T11:05:14.546 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval local_cluster=cluster1 2023-10-11T11:05:14.546 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ local_cluster=cluster1 2023-10-11T11:05:14.546 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval instance=1 2023-10-11T11:05:14.546 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ instance=1 2023-10-11T11:05:14.547 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-10-11T11:05:14.557 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T11:05:14.557 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T11:05:14.557 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T11:05:14.557 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_19 2023-10-11T11:05:14.704 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ return 0 2023-10-11T11:05:14.704 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ status_result=' "state": "Replaying"' 2023-10-11T11:05:14.705 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ echo ' "state": "Replaying"' 2023-10-11T11:05:14.705 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ grep -i Replaying 2023-10-11T11:05:14.706 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: "state": "Replaying" 2023-10-11T11:05:14.709 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ current_state=started 2023-10-11T11:05:14.709 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test started = started 2023-10-11T11:05:14.710 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:05:14.710 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_19 2023-10-11T11:05:14.710 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local local_cluster=cluster1 2023-10-11T11:05:14.710 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:05:14.711 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:05:14.711 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_19 2023-10-11T11:05:14.711 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' snapshot = journal ']' 2023-10-11T11:05:14.711 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' snapshot = snapshot ']' 2023-10-11T11:05:14.712 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_19 2023-10-11T11:05:14.712 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local local_cluster=cluster1 2023-10-11T11:05:14.712 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:05:14.712 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:05:14.713 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_19 2023-10-11T11:05:14.713 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ mkfname cluster2-mirror-image_19.status 2023-10-11T11:05:14.713 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ echo cluster2-mirror-image_19.status 2023-10-11T11:05:14.713 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -e 's|[/ ]|_|g' 2023-10-11T11:05:14.714 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local status_log=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_19.status 2023-10-11T11:05:14.714 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ mkfname cluster1-mirror-image_19.status 2023-10-11T11:05:14.714 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ echo cluster1-mirror-image_19.status 2023-10-11T11:05:14.714 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -e 's|[/ ]|_|g' 2023-10-11T11:05:14.715 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local local_status_log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_19.status 2023-10-11T11:05:14.715 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ mirror_image_snapshot cluster2 mirror image_19 2023-10-11T11:05:14.715 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:05:14.716 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:05:14.716 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_19 2023-10-11T11:05:14.716 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_19 2023-10-11T11:05:15.411 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:Snapshot ID: 155 2023-10-11T11:05:15.422 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_19 /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_19.status 2023-10-11T11:05:15.423 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:05:15.423 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:05:15.423 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_19 2023-10-11T11:05:15.424 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_19.status 2023-10-11T11:05:15.424 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_19 --format xml 2023-10-11T11:05:15.424 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:05:15.465 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-10-11T11:05:15.467 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snapshot_id=155 2023-10-11T11:05:15.467 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T11:05:15.468 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T11:05:15.468 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.2 2023-10-11T11:05:15.668 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_19 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_19.status 2023-10-11T11:05:15.669 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:05:15.669 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:05:15.669 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_19 2023-10-11T11:05:15.669 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_19.status 2023-10-11T11:05:15.670 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_19 --format xml 2023-10-11T11:05:15.670 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:05:15.715 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T11:05:15.717 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id=86 2023-10-11T11:05:15.717 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 155 = 86 2023-10-11T11:05:15.718 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T11:05:15.718 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.4 2023-10-11T11:05:16.119 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_19 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_19.status 2023-10-11T11:05:16.119 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:05:16.119 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:05:16.120 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_19 2023-10-11T11:05:16.120 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_19.status 2023-10-11T11:05:16.120 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_19 --format xml 2023-10-11T11:05:16.120 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:05:16.164 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T11:05:16.164 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T11:05:16.165 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:-:1.1: Document is empty 2023-10-11T11:05:16.166 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: 2023-10-11T11:05:16.166 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:^ 2023-10-11T11:05:16.166 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id= 2023-10-11T11:05:16.167 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 155 = '' 2023-10-11T11:05:16.167 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T11:05:16.167 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.8 2023-10-11T11:05:16.967 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_19 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_19.status 2023-10-11T11:05:16.967 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:05:16.967 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:05:16.968 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_19 2023-10-11T11:05:16.968 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_19.status 2023-10-11T11:05:16.978 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_19 --format xml 2023-10-11T11:05:16.978 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:05:17.070 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T11:05:17.070 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T11:05:17.071 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:-:1.1: Document is empty 2023-10-11T11:05:17.071 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: 2023-10-11T11:05:17.072 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:^ 2023-10-11T11:05:17.072 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id= 2023-10-11T11:05:17.073 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 155 = '' 2023-10-11T11:05:17.073 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T11:05:17.073 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 1.6 2023-10-11T11:05:18.673 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_19 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_19.status 2023-10-11T11:05:18.674 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:05:18.674 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:05:18.674 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_19 2023-10-11T11:05:18.674 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_19.status 2023-10-11T11:05:18.675 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_19 --format xml 2023-10-11T11:05:18.675 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:05:18.725 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T11:05:18.728 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id=155 2023-10-11T11:05:18.728 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 155 = 155 2023-10-11T11:05:18.728 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:05:18.728 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_snap_present cluster1 mirror image_19 snap 2023-10-11T11:05:18.729 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:05:18.729 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:05:18.729 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_19 2023-10-11T11:05:18.729 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap 2023-10-11T11:05:18.729 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T11:05:18.730 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-10-11T11:05:18.730 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 1 2023-10-11T11:05:19.728 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 info mirror/image_19@snap 2023-10-11T11:05:19.770 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:rbd image 'image_19': 2023-10-11T11:05:19.771 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: size 128 MiB in 32 objects 2023-10-11T11:05:19.771 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: order 22 (4 MiB objects) 2023-10-11T11:05:19.771 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: snapshot_count: 2 2023-10-11T11:05:19.771 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: id: 123d3de2d6d5 2023-10-11T11:05:19.771 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: block_name_prefix: rbd_data.123d3de2d6d5 2023-10-11T11:05:19.772 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: format: 2 2023-10-11T11:05:19.772 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-10-11T11:05:19.772 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: op_features: 2023-10-11T11:05:19.772 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: flags: 2023-10-11T11:05:19.773 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: create_timestamp: Wed Oct 11 10:57:23 2023 2023-10-11T11:05:19.773 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: access_timestamp: Wed Oct 11 10:57:23 2023 2023-10-11T11:05:19.773 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: modify_timestamp: Wed Oct 11 10:57:23 2023 2023-10-11T11:05:19.773 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: protected: False 2023-10-11T11:05:19.773 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring state: enabled 2023-10-11T11:05:19.774 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring mode: snapshot 2023-10-11T11:05:19.774 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring global id: 0f059703-71ab-45c9-bae3-1fd34ca31402 2023-10-11T11:05:19.774 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring primary: false 2023-10-11T11:05:19.775 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:05:19.775 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ compare_image_snaps mirror image_19 snap 2023-10-11T11:05:19.776 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:05:19.776 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_19 2023-10-11T11:05:19.776 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap 2023-10-11T11:05:19.776 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local ret=0 2023-10-11T11:05:19.777 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local rmt_export=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_19.export 2023-10-11T11:05:19.777 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local loc_export=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_19.export 2023-10-11T11:05:19.777 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rm -f /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_19.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_19.export 2023-10-11T11:05:19.777 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 -p mirror export image_19@snap /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_19.export 2023-10-11T11:05:19.953 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-10-11T11:05:19.958 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 -p mirror export image_19@snap /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_19.export 2023-10-11T11:05:20.197 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-10-11T11:05:20.204 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ cmp /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_19.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_19.export 2023-10-11T11:05:20.259 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rm -f /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_19.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_19.export 2023-10-11T11:05:20.295 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:05:20.295 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T11:05:20.296 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_20 2023-10-11T11:05:20.296 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ create_snap cluster2 mirror image_20 snap 2023-10-11T11:05:20.296 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:05:20.296 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:05:20.297 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_20 2023-10-11T11:05:20.297 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap 2023-10-11T11:05:20.297 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 -p mirror snap create image_20@snap --debug-rbd=20 --debug-journaler=20 2023-10-11T11:05:21.496 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_replay_started cluster1 mirror image_20 2023-10-11T11:05:21.496 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:05:21.496 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:05:21.496 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_20 2023-10-11T11:05:21.497 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_replay_state cluster1 mirror image_20 started 2023-10-11T11:05:21.497 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:05:21.497 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:05:21.497 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_20 2023-10-11T11:05:21.497 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local state=started 2023-10-11T11:05:21.498 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T11:05:21.498 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-10-11T11:05:21.498 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 1 2023-10-11T11:05:22.497 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test_image_replay_state cluster1 mirror image_20 started 2023-10-11T11:05:22.497 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:05:22.497 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:05:22.498 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_20 2023-10-11T11:05:22.498 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local test_state=started 2023-10-11T11:05:22.498 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local status_result 2023-10-11T11:05:22.498 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_state=stopped 2023-10-11T11:05:22.503 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_20 2023-10-11T11:05:22.503 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ grep -i state 2023-10-11T11:05:22.503 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_instance=cluster1 2023-10-11T11:05:22.503 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T11:05:22.504 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:05:22.504 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance=cluster1 2023-10-11T11:05:22.504 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local loop_instance 2023-10-11T11:05:22.504 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-10-11T11:05:22.504 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sleep 0 2023-10-11T11:05:22.505 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-10-11T11:05:22.505 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ seq 0 3 2023-10-11T11:05:22.505 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-10-11T11:05:22.505 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_20 2023-10-11T11:05:22.505 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1:0 2023-10-11T11:05:22.506 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T11:05:22.506 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance 2023-10-11T11:05:22.506 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-10-11T11:05:22.506 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local val=cluster1:0 2023-10-11T11:05:22.507 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_var_name=cluster 2023-10-11T11:05:22.507 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance_var_name=instance 2023-10-11T11:05:22.507 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ cluster=cluster1 2023-10-11T11:05:22.507 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ instance=0 2023-10-11T11:05:22.507 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' 0 = cluster1:0 ']' 2023-10-11T11:05:22.508 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval cluster=cluster1 2023-10-11T11:05:22.508 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:05:22.508 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval instance=0 2023-10-11T11:05:22.508 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=0 2023-10-11T11:05:22.508 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-10-11T11:05:22.509 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local local_cluster=cluster1:0 2023-10-11T11:05:22.509 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster=cluster1 2023-10-11T11:05:22.509 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance 2023-10-11T11:05:22.509 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-10-11T11:05:22.509 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local val=cluster1:0 2023-10-11T11:05:22.510 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster_var_name=local_cluster 2023-10-11T11:05:22.510 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance_var_name=instance 2023-10-11T11:05:22.510 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:05:22.510 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=0 2023-10-11T11:05:22.510 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ '[' 0 = cluster1:0 ']' 2023-10-11T11:05:22.511 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval local_cluster=cluster1 2023-10-11T11:05:22.511 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ local_cluster=cluster1 2023-10-11T11:05:22.511 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval instance=0 2023-10-11T11:05:22.511 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ instance=0 2023-10-11T11:05:22.511 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-10-11T11:05:22.523 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T11:05:22.524 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T11:05:22.524 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T11:05:22.524 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_20 2023-10-11T11:05:22.623 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:Invalid command: unused arguments: ['mirror/image_20'] 2023-10-11T11:05:22.624 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd mirror status : get status for rbd mirror 2023-10-11T11:05:22.624 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:admin_socket: invalid command 2023-10-11T11:05:22.627 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-10-11T11:05:22.627 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_20 2023-10-11T11:05:22.627 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1:1 2023-10-11T11:05:22.627 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T11:05:22.628 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance 2023-10-11T11:05:22.628 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-10-11T11:05:22.628 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local val=cluster1:1 2023-10-11T11:05:22.628 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_var_name=cluster 2023-10-11T11:05:22.628 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance_var_name=instance 2023-10-11T11:05:22.629 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ cluster=cluster1 2023-10-11T11:05:22.629 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ instance=1 2023-10-11T11:05:22.629 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' 1 = cluster1:1 ']' 2023-10-11T11:05:22.629 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval cluster=cluster1 2023-10-11T11:05:22.629 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:05:22.630 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval instance=1 2023-10-11T11:05:22.630 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=1 2023-10-11T11:05:22.630 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-10-11T11:05:22.630 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local local_cluster=cluster1:1 2023-10-11T11:05:22.630 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster=cluster1 2023-10-11T11:05:22.630 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance 2023-10-11T11:05:22.631 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-10-11T11:05:22.631 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local val=cluster1:1 2023-10-11T11:05:22.631 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster_var_name=local_cluster 2023-10-11T11:05:22.631 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance_var_name=instance 2023-10-11T11:05:22.631 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:05:22.632 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=1 2023-10-11T11:05:22.632 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ '[' 1 = cluster1:1 ']' 2023-10-11T11:05:22.632 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval local_cluster=cluster1 2023-10-11T11:05:22.632 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ local_cluster=cluster1 2023-10-11T11:05:22.632 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval instance=1 2023-10-11T11:05:22.633 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ instance=1 2023-10-11T11:05:22.633 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-10-11T11:05:22.647 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T11:05:22.647 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T11:05:22.648 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T11:05:22.648 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_20 2023-10-11T11:05:22.793 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ return 0 2023-10-11T11:05:22.794 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ status_result=' "state": "Replaying"' 2023-10-11T11:05:22.794 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ echo ' "state": "Replaying"' 2023-10-11T11:05:22.795 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ grep -i Replaying 2023-10-11T11:05:22.795 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: "state": "Replaying" 2023-10-11T11:05:22.797 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ current_state=started 2023-10-11T11:05:22.797 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test started = started 2023-10-11T11:05:22.797 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:05:22.797 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_20 2023-10-11T11:05:22.798 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local local_cluster=cluster1 2023-10-11T11:05:22.798 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:05:22.798 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:05:22.798 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_20 2023-10-11T11:05:22.798 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' snapshot = journal ']' 2023-10-11T11:05:22.799 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' snapshot = snapshot ']' 2023-10-11T11:05:22.799 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_20 2023-10-11T11:05:22.799 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local local_cluster=cluster1 2023-10-11T11:05:22.799 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:05:22.799 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:05:22.800 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_20 2023-10-11T11:05:22.800 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ mkfname cluster2-mirror-image_20.status 2023-10-11T11:05:22.800 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ echo cluster2-mirror-image_20.status 2023-10-11T11:05:22.800 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -e 's|[/ ]|_|g' 2023-10-11T11:05:22.800 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local status_log=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_20.status 2023-10-11T11:05:22.800 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ mkfname cluster1-mirror-image_20.status 2023-10-11T11:05:22.802 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ echo cluster1-mirror-image_20.status 2023-10-11T11:05:22.802 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -e 's|[/ ]|_|g' 2023-10-11T11:05:22.802 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local local_status_log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_20.status 2023-10-11T11:05:22.802 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ mirror_image_snapshot cluster2 mirror image_20 2023-10-11T11:05:22.802 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:05:22.803 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:05:22.803 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_20 2023-10-11T11:05:22.803 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_20 2023-10-11T11:05:23.509 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:Snapshot ID: 157 2023-10-11T11:05:23.521 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_20 /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_20.status 2023-10-11T11:05:23.521 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:05:23.521 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:05:23.521 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_20 2023-10-11T11:05:23.521 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_20.status 2023-10-11T11:05:23.522 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_20 --format xml 2023-10-11T11:05:23.522 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:05:23.566 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-10-11T11:05:23.568 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snapshot_id=157 2023-10-11T11:05:23.569 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T11:05:23.569 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T11:05:23.569 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.2 2023-10-11T11:05:23.770 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_20 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_20.status 2023-10-11T11:05:23.770 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:05:23.770 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:05:23.771 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_20 2023-10-11T11:05:23.771 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_20.status 2023-10-11T11:05:23.771 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_20 --format xml 2023-10-11T11:05:23.771 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:05:23.819 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T11:05:23.820 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T11:05:23.821 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:-:1.1: Document is empty 2023-10-11T11:05:23.822 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: 2023-10-11T11:05:23.822 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:^ 2023-10-11T11:05:23.822 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id= 2023-10-11T11:05:23.822 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 157 = '' 2023-10-11T11:05:23.823 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T11:05:23.823 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.4 2023-10-11T11:05:24.223 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_20 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_20.status 2023-10-11T11:05:24.223 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:05:24.224 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:05:24.224 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_20 2023-10-11T11:05:24.224 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_20.status 2023-10-11T11:05:24.225 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_20 --format xml 2023-10-11T11:05:24.225 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:05:24.271 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T11:05:24.272 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T11:05:24.273 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:-:1.1: Document is empty 2023-10-11T11:05:24.274 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: 2023-10-11T11:05:24.274 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:^ 2023-10-11T11:05:24.274 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id= 2023-10-11T11:05:24.274 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 157 = '' 2023-10-11T11:05:24.275 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T11:05:24.275 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.8 2023-10-11T11:05:25.075 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_20 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_20.status 2023-10-11T11:05:25.075 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:05:25.076 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:05:25.076 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_20 2023-10-11T11:05:25.076 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_20.status 2023-10-11T11:05:25.077 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_20 --format xml 2023-10-11T11:05:25.077 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:05:25.119 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T11:05:25.121 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id=157 2023-10-11T11:05:25.122 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 157 = 157 2023-10-11T11:05:25.122 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:05:25.122 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_snap_present cluster1 mirror image_20 snap 2023-10-11T11:05:25.123 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:05:25.123 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:05:25.123 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_20 2023-10-11T11:05:25.123 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap 2023-10-11T11:05:25.124 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T11:05:25.124 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-10-11T11:05:25.124 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 1 2023-10-11T11:05:26.121 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 info mirror/image_20@snap 2023-10-11T11:05:26.172 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:rbd image 'image_20': 2023-10-11T11:05:26.172 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: size 128 MiB in 32 objects 2023-10-11T11:05:26.172 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: order 22 (4 MiB objects) 2023-10-11T11:05:26.172 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: snapshot_count: 2 2023-10-11T11:05:26.172 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: id: 123dec683176 2023-10-11T11:05:26.173 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: block_name_prefix: rbd_data.123dec683176 2023-10-11T11:05:26.173 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: format: 2 2023-10-11T11:05:26.173 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-10-11T11:05:26.173 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: op_features: 2023-10-11T11:05:26.173 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: flags: 2023-10-11T11:05:26.173 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: create_timestamp: Wed Oct 11 10:57:29 2023 2023-10-11T11:05:26.174 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: access_timestamp: Wed Oct 11 10:57:29 2023 2023-10-11T11:05:26.174 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: modify_timestamp: Wed Oct 11 10:57:29 2023 2023-10-11T11:05:26.174 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: protected: False 2023-10-11T11:05:26.174 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring state: enabled 2023-10-11T11:05:26.174 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring mode: snapshot 2023-10-11T11:05:26.175 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring global id: 46b9e65d-4379-44aa-a3c7-83f3b477d8e6 2023-10-11T11:05:26.175 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring primary: false 2023-10-11T11:05:26.176 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:05:26.176 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ compare_image_snaps mirror image_20 snap 2023-10-11T11:05:26.176 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:05:26.176 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_20 2023-10-11T11:05:26.177 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap 2023-10-11T11:05:26.177 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local ret=0 2023-10-11T11:05:26.177 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local rmt_export=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_20.export 2023-10-11T11:05:26.177 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local loc_export=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_20.export 2023-10-11T11:05:26.177 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rm -f /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_20.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_20.export 2023-10-11T11:05:26.178 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 -p mirror export image_20@snap /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_20.export 2023-10-11T11:05:26.342 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-10-11T11:05:26.349 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 -p mirror export image_20@snap /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_20.export 2023-10-11T11:05:26.564 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-10-11T11:05:26.571 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ cmp /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_20.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_20.export 2023-10-11T11:05:26.625 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rm -f /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_20.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_20.export 2023-10-11T11:05:26.662 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:05:26.662 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T11:05:26.663 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_21 2023-10-11T11:05:26.663 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ create_snap cluster2 mirror image_21 snap 2023-10-11T11:05:26.663 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:05:26.663 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:05:26.664 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_21 2023-10-11T11:05:26.664 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap 2023-10-11T11:05:26.664 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 -p mirror snap create image_21@snap --debug-rbd=20 --debug-journaler=20 2023-10-11T11:05:28.523 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_replay_started cluster1 mirror image_21 2023-10-11T11:05:28.523 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:05:28.523 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:05:28.524 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_21 2023-10-11T11:05:28.524 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_replay_state cluster1 mirror image_21 started 2023-10-11T11:05:28.524 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:05:28.524 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:05:28.525 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_21 2023-10-11T11:05:28.525 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local state=started 2023-10-11T11:05:28.525 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T11:05:28.525 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-10-11T11:05:28.526 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 1 2023-10-11T11:05:29.523 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test_image_replay_state cluster1 mirror image_21 started 2023-10-11T11:05:29.523 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:05:29.524 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:05:29.524 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_21 2023-10-11T11:05:29.524 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local test_state=started 2023-10-11T11:05:29.524 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local status_result 2023-10-11T11:05:29.524 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_state=stopped 2023-10-11T11:05:29.526 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_21 2023-10-11T11:05:29.526 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_instance=cluster1 2023-10-11T11:05:29.526 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ grep -i state 2023-10-11T11:05:29.527 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T11:05:29.527 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:05:29.527 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance=cluster1 2023-10-11T11:05:29.527 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local loop_instance 2023-10-11T11:05:29.527 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-10-11T11:05:29.528 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sleep 0 2023-10-11T11:05:29.528 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-10-11T11:05:29.528 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ seq 0 3 2023-10-11T11:05:29.530 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-10-11T11:05:29.530 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_21 2023-10-11T11:05:29.530 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1:0 2023-10-11T11:05:29.530 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T11:05:29.531 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance 2023-10-11T11:05:29.531 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-10-11T11:05:29.531 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local val=cluster1:0 2023-10-11T11:05:29.531 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_var_name=cluster 2023-10-11T11:05:29.531 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance_var_name=instance 2023-10-11T11:05:29.532 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ cluster=cluster1 2023-10-11T11:05:29.532 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ instance=0 2023-10-11T11:05:29.532 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' 0 = cluster1:0 ']' 2023-10-11T11:05:29.532 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval cluster=cluster1 2023-10-11T11:05:29.532 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:05:29.533 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval instance=0 2023-10-11T11:05:29.533 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=0 2023-10-11T11:05:29.533 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-10-11T11:05:29.533 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local local_cluster=cluster1:0 2023-10-11T11:05:29.535 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster=cluster1 2023-10-11T11:05:29.535 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance 2023-10-11T11:05:29.535 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-10-11T11:05:29.535 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local val=cluster1:0 2023-10-11T11:05:29.535 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster_var_name=local_cluster 2023-10-11T11:05:29.536 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance_var_name=instance 2023-10-11T11:05:29.536 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:05:29.536 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=0 2023-10-11T11:05:29.536 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ '[' 0 = cluster1:0 ']' 2023-10-11T11:05:29.536 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval local_cluster=cluster1 2023-10-11T11:05:29.537 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ local_cluster=cluster1 2023-10-11T11:05:29.537 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval instance=0 2023-10-11T11:05:29.537 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ instance=0 2023-10-11T11:05:29.537 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-10-11T11:05:29.552 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T11:05:29.552 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T11:05:29.553 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T11:05:29.553 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_21 2023-10-11T11:05:29.653 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:Invalid command: unused arguments: ['mirror/image_21'] 2023-10-11T11:05:29.653 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd mirror status : get status for rbd mirror 2023-10-11T11:05:29.654 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:admin_socket: invalid command 2023-10-11T11:05:29.656 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-10-11T11:05:29.657 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_21 2023-10-11T11:05:29.657 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1:1 2023-10-11T11:05:29.657 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T11:05:29.657 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance 2023-10-11T11:05:29.657 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-10-11T11:05:29.658 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local val=cluster1:1 2023-10-11T11:05:29.658 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_var_name=cluster 2023-10-11T11:05:29.658 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance_var_name=instance 2023-10-11T11:05:29.658 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ cluster=cluster1 2023-10-11T11:05:29.658 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ instance=1 2023-10-11T11:05:29.659 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' 1 = cluster1:1 ']' 2023-10-11T11:05:29.659 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval cluster=cluster1 2023-10-11T11:05:29.659 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:05:29.659 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval instance=1 2023-10-11T11:05:29.659 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=1 2023-10-11T11:05:29.659 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-10-11T11:05:29.660 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local local_cluster=cluster1:1 2023-10-11T11:05:29.660 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster=cluster1 2023-10-11T11:05:29.660 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance 2023-10-11T11:05:29.660 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-10-11T11:05:29.660 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local val=cluster1:1 2023-10-11T11:05:29.661 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster_var_name=local_cluster 2023-10-11T11:05:29.661 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance_var_name=instance 2023-10-11T11:05:29.661 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:05:29.661 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=1 2023-10-11T11:05:29.661 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ '[' 1 = cluster1:1 ']' 2023-10-11T11:05:29.662 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval local_cluster=cluster1 2023-10-11T11:05:29.662 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ local_cluster=cluster1 2023-10-11T11:05:29.662 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval instance=1 2023-10-11T11:05:29.662 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ instance=1 2023-10-11T11:05:29.662 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-10-11T11:05:29.676 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T11:05:29.676 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T11:05:29.676 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T11:05:29.676 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_21 2023-10-11T11:05:29.828 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ return 0 2023-10-11T11:05:29.829 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ status_result=' "state": "Replaying"' 2023-10-11T11:05:29.829 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ echo ' "state": "Replaying"' 2023-10-11T11:05:29.829 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ grep -i Replaying 2023-10-11T11:05:29.830 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: "state": "Replaying" 2023-10-11T11:05:29.832 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ current_state=started 2023-10-11T11:05:29.832 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test started = started 2023-10-11T11:05:29.833 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:05:29.833 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_21 2023-10-11T11:05:29.833 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local local_cluster=cluster1 2023-10-11T11:05:29.833 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:05:29.833 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:05:29.834 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_21 2023-10-11T11:05:29.834 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' snapshot = journal ']' 2023-10-11T11:05:29.834 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' snapshot = snapshot ']' 2023-10-11T11:05:29.834 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_21 2023-10-11T11:05:29.835 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local local_cluster=cluster1 2023-10-11T11:05:29.835 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:05:29.835 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:05:29.835 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_21 2023-10-11T11:05:29.835 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ mkfname cluster2-mirror-image_21.status 2023-10-11T11:05:29.836 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ echo cluster2-mirror-image_21.status 2023-10-11T11:05:29.836 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -e 's|[/ ]|_|g' 2023-10-11T11:05:29.836 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local status_log=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_21.status 2023-10-11T11:05:29.836 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ mkfname cluster1-mirror-image_21.status 2023-10-11T11:05:29.836 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ echo cluster1-mirror-image_21.status 2023-10-11T11:05:29.837 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -e 's|[/ ]|_|g' 2023-10-11T11:05:29.838 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local local_status_log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_21.status 2023-10-11T11:05:29.838 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ mirror_image_snapshot cluster2 mirror image_21 2023-10-11T11:05:29.838 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:05:29.838 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:05:29.839 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_21 2023-10-11T11:05:29.839 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_21 2023-10-11T11:05:30.521 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:Snapshot ID: 159 2023-10-11T11:05:30.531 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_21 /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_21.status 2023-10-11T11:05:30.531 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:05:30.532 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:05:30.532 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_21 2023-10-11T11:05:30.532 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_21.status 2023-10-11T11:05:30.533 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_21 --format xml 2023-10-11T11:05:30.533 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:05:30.574 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-10-11T11:05:30.576 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snapshot_id=159 2023-10-11T11:05:30.576 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T11:05:30.576 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T11:05:30.576 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.2 2023-10-11T11:05:30.777 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_21 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_21.status 2023-10-11T11:05:30.777 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:05:30.778 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:05:30.778 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_21 2023-10-11T11:05:30.778 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_21.status 2023-10-11T11:05:30.778 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_21 --format xml 2023-10-11T11:05:30.779 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:05:30.822 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T11:05:30.822 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T11:05:30.824 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:-:1.1: Document is empty 2023-10-11T11:05:30.824 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: 2023-10-11T11:05:30.824 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:^ 2023-10-11T11:05:30.825 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id= 2023-10-11T11:05:30.825 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 159 = '' 2023-10-11T11:05:30.825 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T11:05:30.825 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.4 2023-10-11T11:05:31.225 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_21 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_21.status 2023-10-11T11:05:31.225 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:05:31.225 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:05:31.226 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_21 2023-10-11T11:05:31.226 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_21.status 2023-10-11T11:05:31.226 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_21 --format xml 2023-10-11T11:05:31.226 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:05:31.272 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T11:05:31.272 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T11:05:31.273 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:-:1.1: Document is empty 2023-10-11T11:05:31.274 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: 2023-10-11T11:05:31.274 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:^ 2023-10-11T11:05:31.274 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id= 2023-10-11T11:05:31.275 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 159 = '' 2023-10-11T11:05:31.275 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T11:05:31.275 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.8 2023-10-11T11:05:32.075 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_21 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_21.status 2023-10-11T11:05:32.075 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:05:32.075 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:05:32.076 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_21 2023-10-11T11:05:32.076 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_21.status 2023-10-11T11:05:32.076 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_21 --format xml 2023-10-11T11:05:32.076 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:05:32.120 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T11:05:32.122 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id=159 2023-10-11T11:05:32.122 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 159 = 159 2023-10-11T11:05:32.123 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:05:32.123 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_snap_present cluster1 mirror image_21 snap 2023-10-11T11:05:32.123 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:05:32.123 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:05:32.123 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_21 2023-10-11T11:05:32.124 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap 2023-10-11T11:05:32.124 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T11:05:32.124 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-10-11T11:05:32.124 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 1 2023-10-11T11:05:33.123 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 info mirror/image_21@snap 2023-10-11T11:05:33.373 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:rbd image 'image_21': 2023-10-11T11:05:33.374 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: size 128 MiB in 32 objects 2023-10-11T11:05:33.374 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: order 22 (4 MiB objects) 2023-10-11T11:05:33.374 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: snapshot_count: 2 2023-10-11T11:05:33.374 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: id: 123dc04460e 2023-10-11T11:05:33.375 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: block_name_prefix: rbd_data.123dc04460e 2023-10-11T11:05:33.375 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: format: 2 2023-10-11T11:05:33.375 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-10-11T11:05:33.375 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: op_features: 2023-10-11T11:05:33.375 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: flags: 2023-10-11T11:05:33.376 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: create_timestamp: Wed Oct 11 10:57:38 2023 2023-10-11T11:05:33.376 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: access_timestamp: Wed Oct 11 10:57:38 2023 2023-10-11T11:05:33.376 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: modify_timestamp: Wed Oct 11 10:57:38 2023 2023-10-11T11:05:33.376 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: protected: False 2023-10-11T11:05:33.376 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring state: enabled 2023-10-11T11:05:33.377 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring mode: snapshot 2023-10-11T11:05:33.377 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring global id: 11e7aef5-a626-4f47-9643-5ab9888eea8f 2023-10-11T11:05:33.377 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring primary: false 2023-10-11T11:05:33.378 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:05:33.378 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ compare_image_snaps mirror image_21 snap 2023-10-11T11:05:33.379 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:05:33.379 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_21 2023-10-11T11:05:33.379 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap 2023-10-11T11:05:33.379 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local ret=0 2023-10-11T11:05:33.379 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local rmt_export=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_21.export 2023-10-11T11:05:33.380 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local loc_export=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_21.export 2023-10-11T11:05:33.380 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rm -f /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_21.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_21.export 2023-10-11T11:05:33.380 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 -p mirror export image_21@snap /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_21.export 2023-10-11T11:05:33.567 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-10-11T11:05:33.572 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 -p mirror export image_21@snap /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_21.export 2023-10-11T11:05:33.841 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-10-11T11:05:33.848 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ cmp /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_21.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_21.export 2023-10-11T11:05:33.902 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rm -f /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_21.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_21.export 2023-10-11T11:05:33.938 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:05:33.938 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T11:05:33.938 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_22 2023-10-11T11:05:33.939 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ create_snap cluster2 mirror image_22 snap 2023-10-11T11:05:33.939 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:05:33.939 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:05:33.939 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_22 2023-10-11T11:05:33.940 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap 2023-10-11T11:05:33.940 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 -p mirror snap create image_22@snap --debug-rbd=20 --debug-journaler=20 2023-10-11T11:05:35.544 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_replay_started cluster1 mirror image_22 2023-10-11T11:05:35.544 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:05:35.544 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:05:35.544 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_22 2023-10-11T11:05:35.545 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_replay_state cluster1 mirror image_22 started 2023-10-11T11:05:35.545 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:05:35.545 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:05:35.545 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_22 2023-10-11T11:05:35.545 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local state=started 2023-10-11T11:05:35.545 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T11:05:35.546 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-10-11T11:05:35.546 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 1 2023-10-11T11:05:36.545 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test_image_replay_state cluster1 mirror image_22 started 2023-10-11T11:05:36.545 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:05:36.545 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:05:36.545 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_22 2023-10-11T11:05:36.546 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local test_state=started 2023-10-11T11:05:36.546 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local status_result 2023-10-11T11:05:36.546 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_state=stopped 2023-10-11T11:05:36.547 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_22 2023-10-11T11:05:36.547 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_instance=cluster1 2023-10-11T11:05:36.548 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T11:05:36.548 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:05:36.548 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ grep -i state 2023-10-11T11:05:36.548 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance=cluster1 2023-10-11T11:05:36.548 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local loop_instance 2023-10-11T11:05:36.548 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-10-11T11:05:36.549 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sleep 0 2023-10-11T11:05:36.549 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-10-11T11:05:36.549 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ seq 0 3 2023-10-11T11:05:36.550 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-10-11T11:05:36.551 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_22 2023-10-11T11:05:36.551 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1:0 2023-10-11T11:05:36.551 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T11:05:36.551 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance 2023-10-11T11:05:36.551 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-10-11T11:05:36.552 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local val=cluster1:0 2023-10-11T11:05:36.552 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_var_name=cluster 2023-10-11T11:05:36.552 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance_var_name=instance 2023-10-11T11:05:36.552 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ cluster=cluster1 2023-10-11T11:05:36.552 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ instance=0 2023-10-11T11:05:36.552 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' 0 = cluster1:0 ']' 2023-10-11T11:05:36.553 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval cluster=cluster1 2023-10-11T11:05:36.553 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:05:36.553 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval instance=0 2023-10-11T11:05:36.553 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=0 2023-10-11T11:05:36.555 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-10-11T11:05:36.555 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local local_cluster=cluster1:0 2023-10-11T11:05:36.555 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster=cluster1 2023-10-11T11:05:36.555 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance 2023-10-11T11:05:36.555 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-10-11T11:05:36.556 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local val=cluster1:0 2023-10-11T11:05:36.556 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster_var_name=local_cluster 2023-10-11T11:05:36.556 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance_var_name=instance 2023-10-11T11:05:36.556 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:05:36.556 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=0 2023-10-11T11:05:36.556 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ '[' 0 = cluster1:0 ']' 2023-10-11T11:05:36.557 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval local_cluster=cluster1 2023-10-11T11:05:36.557 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ local_cluster=cluster1 2023-10-11T11:05:36.557 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval instance=0 2023-10-11T11:05:36.557 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ instance=0 2023-10-11T11:05:36.557 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-10-11T11:05:36.574 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T11:05:36.574 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T11:05:36.574 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T11:05:36.574 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_22 2023-10-11T11:05:36.675 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:Invalid command: unused arguments: ['mirror/image_22'] 2023-10-11T11:05:36.675 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd mirror status : get status for rbd mirror 2023-10-11T11:05:36.675 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:admin_socket: invalid command 2023-10-11T11:05:36.678 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-10-11T11:05:36.678 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_22 2023-10-11T11:05:36.679 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1:1 2023-10-11T11:05:36.679 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T11:05:36.679 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance 2023-10-11T11:05:36.679 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-10-11T11:05:36.679 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local val=cluster1:1 2023-10-11T11:05:36.680 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_var_name=cluster 2023-10-11T11:05:36.680 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance_var_name=instance 2023-10-11T11:05:36.680 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ cluster=cluster1 2023-10-11T11:05:36.680 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ instance=1 2023-10-11T11:05:36.680 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' 1 = cluster1:1 ']' 2023-10-11T11:05:36.680 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval cluster=cluster1 2023-10-11T11:05:36.681 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:05:36.681 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval instance=1 2023-10-11T11:05:36.681 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=1 2023-10-11T11:05:36.681 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-10-11T11:05:36.681 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local local_cluster=cluster1:1 2023-10-11T11:05:36.682 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster=cluster1 2023-10-11T11:05:36.682 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance 2023-10-11T11:05:36.682 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-10-11T11:05:36.682 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local val=cluster1:1 2023-10-11T11:05:36.682 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster_var_name=local_cluster 2023-10-11T11:05:36.683 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance_var_name=instance 2023-10-11T11:05:36.683 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:05:36.683 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=1 2023-10-11T11:05:36.683 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ '[' 1 = cluster1:1 ']' 2023-10-11T11:05:36.683 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval local_cluster=cluster1 2023-10-11T11:05:36.683 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ local_cluster=cluster1 2023-10-11T11:05:36.684 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval instance=1 2023-10-11T11:05:36.684 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ instance=1 2023-10-11T11:05:36.684 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-10-11T11:05:36.697 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T11:05:36.698 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T11:05:36.698 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T11:05:36.698 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_22 2023-10-11T11:05:36.845 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ return 0 2023-10-11T11:05:36.845 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ status_result=' "state": "Replaying"' 2023-10-11T11:05:36.846 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ echo ' "state": "Replaying"' 2023-10-11T11:05:36.846 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ grep -i Replaying 2023-10-11T11:05:36.847 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: "state": "Replaying" 2023-10-11T11:05:36.849 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ current_state=started 2023-10-11T11:05:36.849 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test started = started 2023-10-11T11:05:36.849 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:05:36.850 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_22 2023-10-11T11:05:36.850 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local local_cluster=cluster1 2023-10-11T11:05:36.850 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:05:36.850 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:05:36.851 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_22 2023-10-11T11:05:36.851 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' snapshot = journal ']' 2023-10-11T11:05:36.851 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' snapshot = snapshot ']' 2023-10-11T11:05:36.851 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_22 2023-10-11T11:05:36.852 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local local_cluster=cluster1 2023-10-11T11:05:36.852 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:05:36.852 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:05:36.852 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_22 2023-10-11T11:05:36.852 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ mkfname cluster2-mirror-image_22.status 2023-10-11T11:05:36.853 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ echo cluster2-mirror-image_22.status 2023-10-11T11:05:36.853 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -e 's|[/ ]|_|g' 2023-10-11T11:05:36.853 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local status_log=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_22.status 2023-10-11T11:05:36.853 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ mkfname cluster1-mirror-image_22.status 2023-10-11T11:05:36.854 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ echo cluster1-mirror-image_22.status 2023-10-11T11:05:36.854 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -e 's|[/ ]|_|g' 2023-10-11T11:05:36.855 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local local_status_log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_22.status 2023-10-11T11:05:36.855 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ mirror_image_snapshot cluster2 mirror image_22 2023-10-11T11:05:36.855 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:05:36.855 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:05:36.856 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_22 2023-10-11T11:05:36.856 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_22 2023-10-11T11:05:37.548 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:Snapshot ID: 161 2023-10-11T11:05:37.560 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_22 /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_22.status 2023-10-11T11:05:37.561 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:05:37.561 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:05:37.561 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_22 2023-10-11T11:05:37.561 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_22.status 2023-10-11T11:05:37.561 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_22 --format xml 2023-10-11T11:05:37.562 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:05:37.603 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-10-11T11:05:37.605 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snapshot_id=161 2023-10-11T11:05:37.605 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T11:05:37.606 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T11:05:37.606 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.2 2023-10-11T11:05:37.806 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_22 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_22.status 2023-10-11T11:05:37.807 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:05:37.807 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:05:37.807 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_22 2023-10-11T11:05:37.807 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_22.status 2023-10-11T11:05:37.808 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_22 --format xml 2023-10-11T11:05:37.808 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:05:37.852 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T11:05:37.853 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T11:05:37.854 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:-:1.1: Document is empty 2023-10-11T11:05:37.854 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: 2023-10-11T11:05:37.855 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:^ 2023-10-11T11:05:37.855 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id= 2023-10-11T11:05:37.855 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 161 = '' 2023-10-11T11:05:37.855 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T11:05:37.856 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.4 2023-10-11T11:05:38.255 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_22 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_22.status 2023-10-11T11:05:38.255 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:05:38.255 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:05:38.256 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_22 2023-10-11T11:05:38.256 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_22.status 2023-10-11T11:05:38.256 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_22 --format xml 2023-10-11T11:05:38.256 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:05:38.302 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T11:05:38.302 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T11:05:38.303 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:-:1.1: Document is empty 2023-10-11T11:05:38.304 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: 2023-10-11T11:05:38.304 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:^ 2023-10-11T11:05:38.304 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id= 2023-10-11T11:05:38.304 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 161 = '' 2023-10-11T11:05:38.304 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T11:05:38.305 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.8 2023-10-11T11:05:39.105 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_22 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_22.status 2023-10-11T11:05:39.106 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:05:39.106 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:05:39.106 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_22 2023-10-11T11:05:39.107 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_22.status 2023-10-11T11:05:39.107 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_22 --format xml 2023-10-11T11:05:39.107 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:05:39.151 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T11:05:39.153 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id=161 2023-10-11T11:05:39.154 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 161 = 161 2023-10-11T11:05:39.154 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:05:39.154 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_snap_present cluster1 mirror image_22 snap 2023-10-11T11:05:39.155 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:05:39.155 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:05:39.155 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_22 2023-10-11T11:05:39.155 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap 2023-10-11T11:05:39.156 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T11:05:39.156 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-10-11T11:05:39.156 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 1 2023-10-11T11:05:40.154 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 info mirror/image_22@snap 2023-10-11T11:05:40.199 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:rbd image 'image_22': 2023-10-11T11:05:40.199 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: size 128 MiB in 32 objects 2023-10-11T11:05:40.199 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: order 22 (4 MiB objects) 2023-10-11T11:05:40.200 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: snapshot_count: 2 2023-10-11T11:05:40.200 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: id: 123dfac83d3a 2023-10-11T11:05:40.200 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: block_name_prefix: rbd_data.123dfac83d3a 2023-10-11T11:05:40.200 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: format: 2 2023-10-11T11:05:40.201 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-10-11T11:05:40.201 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: op_features: 2023-10-11T11:05:40.201 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: flags: 2023-10-11T11:05:40.201 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: create_timestamp: Wed Oct 11 10:57:49 2023 2023-10-11T11:05:40.202 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: access_timestamp: Wed Oct 11 10:57:49 2023 2023-10-11T11:05:40.202 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: modify_timestamp: Wed Oct 11 10:57:49 2023 2023-10-11T11:05:40.202 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: protected: False 2023-10-11T11:05:40.202 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring state: enabled 2023-10-11T11:05:40.203 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring mode: snapshot 2023-10-11T11:05:40.203 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring global id: 1288fb83-4aad-4242-bdd2-1944a6794332 2023-10-11T11:05:40.203 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring primary: false 2023-10-11T11:05:40.205 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:05:40.205 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ compare_image_snaps mirror image_22 snap 2023-10-11T11:05:40.205 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:05:40.205 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_22 2023-10-11T11:05:40.206 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap 2023-10-11T11:05:40.206 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local ret=0 2023-10-11T11:05:40.206 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local rmt_export=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_22.export 2023-10-11T11:05:40.206 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local loc_export=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_22.export 2023-10-11T11:05:40.207 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rm -f /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_22.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_22.export 2023-10-11T11:05:40.207 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 -p mirror export image_22@snap /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_22.export 2023-10-11T11:05:40.378 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-10-11T11:05:40.384 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 -p mirror export image_22@snap /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_22.export 2023-10-11T11:05:40.614 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-10-11T11:05:40.620 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ cmp /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_22.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_22.export 2023-10-11T11:05:40.675 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rm -f /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_22.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_22.export 2023-10-11T11:05:40.712 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:05:40.712 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T11:05:40.713 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_23 2023-10-11T11:05:40.713 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ create_snap cluster2 mirror image_23 snap 2023-10-11T11:05:40.713 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:05:40.713 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:05:40.713 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_23 2023-10-11T11:05:40.714 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap 2023-10-11T11:05:40.714 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 -p mirror snap create image_23@snap --debug-rbd=20 --debug-journaler=20 2023-10-11T11:05:42.407 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_replay_started cluster1 mirror image_23 2023-10-11T11:05:42.407 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:05:42.408 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:05:42.408 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_23 2023-10-11T11:05:42.408 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_replay_state cluster1 mirror image_23 started 2023-10-11T11:05:42.408 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:05:42.408 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:05:42.409 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_23 2023-10-11T11:05:42.409 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local state=started 2023-10-11T11:05:42.409 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T11:05:42.409 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-10-11T11:05:42.409 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 1 2023-10-11T11:05:43.409 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test_image_replay_state cluster1 mirror image_23 started 2023-10-11T11:05:43.409 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:05:43.409 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:05:43.410 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_23 2023-10-11T11:05:43.410 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local test_state=started 2023-10-11T11:05:43.410 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local status_result 2023-10-11T11:05:43.410 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_state=stopped 2023-10-11T11:05:43.410 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_23 2023-10-11T11:05:43.411 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ grep -i state 2023-10-11T11:05:43.411 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_instance=cluster1 2023-10-11T11:05:43.411 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T11:05:43.411 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:05:43.411 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance=cluster1 2023-10-11T11:05:43.412 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local loop_instance 2023-10-11T11:05:43.412 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-10-11T11:05:43.412 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sleep 0 2023-10-11T11:05:43.414 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-10-11T11:05:43.414 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ seq 0 3 2023-10-11T11:05:43.414 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-10-11T11:05:43.414 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_23 2023-10-11T11:05:43.415 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1:0 2023-10-11T11:05:43.415 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T11:05:43.415 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance 2023-10-11T11:05:43.415 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-10-11T11:05:43.415 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local val=cluster1:0 2023-10-11T11:05:43.415 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_var_name=cluster 2023-10-11T11:05:43.416 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance_var_name=instance 2023-10-11T11:05:43.416 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ cluster=cluster1 2023-10-11T11:05:43.416 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ instance=0 2023-10-11T11:05:43.416 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' 0 = cluster1:0 ']' 2023-10-11T11:05:43.416 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval cluster=cluster1 2023-10-11T11:05:43.416 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:05:43.417 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval instance=0 2023-10-11T11:05:43.417 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=0 2023-10-11T11:05:43.417 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-10-11T11:05:43.417 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local local_cluster=cluster1:0 2023-10-11T11:05:43.417 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster=cluster1 2023-10-11T11:05:43.417 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance 2023-10-11T11:05:43.418 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-10-11T11:05:43.418 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local val=cluster1:0 2023-10-11T11:05:43.418 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster_var_name=local_cluster 2023-10-11T11:05:43.418 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance_var_name=instance 2023-10-11T11:05:43.419 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:05:43.419 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=0 2023-10-11T11:05:43.419 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ '[' 0 = cluster1:0 ']' 2023-10-11T11:05:43.419 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval local_cluster=cluster1 2023-10-11T11:05:43.420 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ local_cluster=cluster1 2023-10-11T11:05:43.420 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval instance=0 2023-10-11T11:05:43.420 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ instance=0 2023-10-11T11:05:43.420 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-10-11T11:05:43.435 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T11:05:43.435 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T11:05:43.436 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T11:05:43.436 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_23 2023-10-11T11:05:43.536 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:Invalid command: unused arguments: ['mirror/image_23'] 2023-10-11T11:05:43.537 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd mirror status : get status for rbd mirror 2023-10-11T11:05:43.537 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:admin_socket: invalid command 2023-10-11T11:05:43.540 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-10-11T11:05:43.540 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_23 2023-10-11T11:05:43.540 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1:1 2023-10-11T11:05:43.541 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T11:05:43.541 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance 2023-10-11T11:05:43.541 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-10-11T11:05:43.541 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local val=cluster1:1 2023-10-11T11:05:43.541 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_var_name=cluster 2023-10-11T11:05:43.542 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance_var_name=instance 2023-10-11T11:05:43.542 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ cluster=cluster1 2023-10-11T11:05:43.542 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ instance=1 2023-10-11T11:05:43.542 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' 1 = cluster1:1 ']' 2023-10-11T11:05:43.542 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval cluster=cluster1 2023-10-11T11:05:43.543 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:05:43.543 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval instance=1 2023-10-11T11:05:43.543 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=1 2023-10-11T11:05:43.543 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-10-11T11:05:43.543 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local local_cluster=cluster1:1 2023-10-11T11:05:43.544 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster=cluster1 2023-10-11T11:05:43.544 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance 2023-10-11T11:05:43.544 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-10-11T11:05:43.544 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local val=cluster1:1 2023-10-11T11:05:43.544 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster_var_name=local_cluster 2023-10-11T11:05:43.544 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance_var_name=instance 2023-10-11T11:05:43.545 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:05:43.545 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=1 2023-10-11T11:05:43.545 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ '[' 1 = cluster1:1 ']' 2023-10-11T11:05:43.545 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval local_cluster=cluster1 2023-10-11T11:05:43.545 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ local_cluster=cluster1 2023-10-11T11:05:43.546 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval instance=1 2023-10-11T11:05:43.546 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ instance=1 2023-10-11T11:05:43.546 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-10-11T11:05:43.560 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T11:05:43.560 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T11:05:43.561 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T11:05:43.561 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_23 2023-10-11T11:05:43.710 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ return 0 2023-10-11T11:05:43.711 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ status_result=' "state": "Replaying"' 2023-10-11T11:05:43.711 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ echo ' "state": "Replaying"' 2023-10-11T11:05:43.711 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ grep -i Replaying 2023-10-11T11:05:43.712 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: "state": "Replaying" 2023-10-11T11:05:43.714 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ current_state=started 2023-10-11T11:05:43.714 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test started = started 2023-10-11T11:05:43.714 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:05:43.714 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_23 2023-10-11T11:05:43.714 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local local_cluster=cluster1 2023-10-11T11:05:43.715 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:05:43.715 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:05:43.715 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_23 2023-10-11T11:05:43.715 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' snapshot = journal ']' 2023-10-11T11:05:43.715 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' snapshot = snapshot ']' 2023-10-11T11:05:43.716 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_23 2023-10-11T11:05:43.716 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local local_cluster=cluster1 2023-10-11T11:05:43.716 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:05:43.716 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:05:43.716 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_23 2023-10-11T11:05:43.716 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ mkfname cluster2-mirror-image_23.status 2023-10-11T11:05:43.717 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ echo cluster2-mirror-image_23.status 2023-10-11T11:05:43.717 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -e 's|[/ ]|_|g' 2023-10-11T11:05:43.718 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local status_log=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_23.status 2023-10-11T11:05:43.718 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ mkfname cluster1-mirror-image_23.status 2023-10-11T11:05:43.718 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ echo cluster1-mirror-image_23.status 2023-10-11T11:05:43.719 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -e 's|[/ ]|_|g' 2023-10-11T11:05:43.719 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local local_status_log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_23.status 2023-10-11T11:05:43.719 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ mirror_image_snapshot cluster2 mirror image_23 2023-10-11T11:05:43.719 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:05:43.719 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:05:43.720 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_23 2023-10-11T11:05:43.720 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_23 2023-10-11T11:05:44.407 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:Snapshot ID: 163 2023-10-11T11:05:44.418 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_23 /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_23.status 2023-10-11T11:05:44.419 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:05:44.419 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:05:44.419 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_23 2023-10-11T11:05:44.419 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_23.status 2023-10-11T11:05:44.420 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_23 --format xml 2023-10-11T11:05:44.420 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:05:44.463 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-10-11T11:05:44.465 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snapshot_id=163 2023-10-11T11:05:44.465 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T11:05:44.465 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T11:05:44.465 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.2 2023-10-11T11:05:44.666 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_23 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_23.status 2023-10-11T11:05:44.666 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:05:44.666 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:05:44.667 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_23 2023-10-11T11:05:44.667 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_23.status 2023-10-11T11:05:44.667 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_23 --format xml 2023-10-11T11:05:44.667 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:05:44.712 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T11:05:44.712 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T11:05:44.713 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:-:1.1: Document is empty 2023-10-11T11:05:44.714 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: 2023-10-11T11:05:44.714 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:^ 2023-10-11T11:05:44.715 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id= 2023-10-11T11:05:44.715 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 163 = '' 2023-10-11T11:05:44.715 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T11:05:44.715 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.4 2023-10-11T11:05:45.115 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_23 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_23.status 2023-10-11T11:05:45.116 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:05:45.116 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:05:45.116 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_23 2023-10-11T11:05:45.116 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_23.status 2023-10-11T11:05:45.117 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_23 --format xml 2023-10-11T11:05:45.117 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:05:45.161 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T11:05:45.162 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T11:05:45.163 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:-:1.1: Document is empty 2023-10-11T11:05:45.163 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: 2023-10-11T11:05:45.163 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:^ 2023-10-11T11:05:45.164 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id= 2023-10-11T11:05:45.164 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 163 = '' 2023-10-11T11:05:45.165 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T11:05:45.165 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.8 2023-10-11T11:05:45.965 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_23 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_23.status 2023-10-11T11:05:45.965 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:05:45.966 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:05:45.966 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_23 2023-10-11T11:05:45.966 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_23.status 2023-10-11T11:05:45.966 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_23 --format xml 2023-10-11T11:05:45.966 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:05:46.012 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T11:05:46.014 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id=163 2023-10-11T11:05:46.014 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 163 = 163 2023-10-11T11:05:46.014 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:05:46.015 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_snap_present cluster1 mirror image_23 snap 2023-10-11T11:05:46.015 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:05:46.015 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:05:46.016 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_23 2023-10-11T11:05:46.016 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap 2023-10-11T11:05:46.016 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T11:05:46.016 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-10-11T11:05:46.016 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 1 2023-10-11T11:05:47.015 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 info mirror/image_23@snap 2023-10-11T11:05:47.061 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:rbd image 'image_23': 2023-10-11T11:05:47.061 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: size 128 MiB in 32 objects 2023-10-11T11:05:47.062 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: order 22 (4 MiB objects) 2023-10-11T11:05:47.062 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: snapshot_count: 2 2023-10-11T11:05:47.062 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: id: 123dbcf547b8 2023-10-11T11:05:47.062 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: block_name_prefix: rbd_data.123dbcf547b8 2023-10-11T11:05:47.062 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: format: 2 2023-10-11T11:05:47.063 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-10-11T11:05:47.063 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: op_features: 2023-10-11T11:05:47.063 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: flags: 2023-10-11T11:05:47.063 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: create_timestamp: Wed Oct 11 10:58:02 2023 2023-10-11T11:05:47.063 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: access_timestamp: Wed Oct 11 10:58:02 2023 2023-10-11T11:05:47.064 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: modify_timestamp: Wed Oct 11 10:58:02 2023 2023-10-11T11:05:47.064 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: protected: False 2023-10-11T11:05:47.064 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring state: enabled 2023-10-11T11:05:47.064 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring mode: snapshot 2023-10-11T11:05:47.065 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring global id: b3738f1e-cf2d-40b8-80a2-69299f4bee7b 2023-10-11T11:05:47.065 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring primary: false 2023-10-11T11:05:47.066 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:05:47.066 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ compare_image_snaps mirror image_23 snap 2023-10-11T11:05:47.066 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:05:47.067 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_23 2023-10-11T11:05:47.067 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap 2023-10-11T11:05:47.067 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local ret=0 2023-10-11T11:05:47.067 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local rmt_export=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_23.export 2023-10-11T11:05:47.067 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local loc_export=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_23.export 2023-10-11T11:05:47.068 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rm -f /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_23.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_23.export 2023-10-11T11:05:47.068 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 -p mirror export image_23@snap /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_23.export 2023-10-11T11:05:47.250 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-10-11T11:05:47.257 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 -p mirror export image_23@snap /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_23.export 2023-10-11T11:05:47.482 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-10-11T11:05:47.488 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ cmp /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_23.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_23.export 2023-10-11T11:05:47.542 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rm -f /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_23.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_23.export 2023-10-11T11:05:47.579 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:05:47.579 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T11:05:47.580 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_24 2023-10-11T11:05:47.580 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ create_snap cluster2 mirror image_24 snap 2023-10-11T11:05:47.580 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:05:47.580 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:05:47.580 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_24 2023-10-11T11:05:47.581 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap 2023-10-11T11:05:47.581 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 -p mirror snap create image_24@snap --debug-rbd=20 --debug-journaler=20 2023-10-11T11:05:49.504 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_replay_started cluster1 mirror image_24 2023-10-11T11:05:49.505 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:05:49.505 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:05:49.505 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_24 2023-10-11T11:05:49.506 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_replay_state cluster1 mirror image_24 started 2023-10-11T11:05:49.506 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:05:49.506 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:05:49.506 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_24 2023-10-11T11:05:49.507 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local state=started 2023-10-11T11:05:49.507 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T11:05:49.507 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-10-11T11:05:49.507 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 1 2023-10-11T11:05:50.505 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test_image_replay_state cluster1 mirror image_24 started 2023-10-11T11:05:50.505 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:05:50.506 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:05:50.506 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_24 2023-10-11T11:05:50.506 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local test_state=started 2023-10-11T11:05:50.506 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local status_result 2023-10-11T11:05:50.507 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_state=stopped 2023-10-11T11:05:50.509 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_24 2023-10-11T11:05:50.510 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_instance=cluster1 2023-10-11T11:05:50.510 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ grep -i state 2023-10-11T11:05:50.510 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T11:05:50.510 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:05:50.511 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance=cluster1 2023-10-11T11:05:50.511 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local loop_instance 2023-10-11T11:05:50.511 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-10-11T11:05:50.511 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sleep 0 2023-10-11T11:05:50.511 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-10-11T11:05:50.512 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ seq 0 3 2023-10-11T11:05:50.512 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-10-11T11:05:50.512 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_24 2023-10-11T11:05:50.512 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1:0 2023-10-11T11:05:50.512 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T11:05:50.513 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance 2023-10-11T11:05:50.513 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-10-11T11:05:50.513 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local val=cluster1:0 2023-10-11T11:05:50.513 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_var_name=cluster 2023-10-11T11:05:50.513 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance_var_name=instance 2023-10-11T11:05:50.514 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ cluster=cluster1 2023-10-11T11:05:50.514 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ instance=0 2023-10-11T11:05:50.514 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' 0 = cluster1:0 ']' 2023-10-11T11:05:50.514 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval cluster=cluster1 2023-10-11T11:05:50.514 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:05:50.515 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval instance=0 2023-10-11T11:05:50.515 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=0 2023-10-11T11:05:50.515 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-10-11T11:05:50.515 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local local_cluster=cluster1:0 2023-10-11T11:05:50.517 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster=cluster1 2023-10-11T11:05:50.517 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance 2023-10-11T11:05:50.517 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-10-11T11:05:50.517 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local val=cluster1:0 2023-10-11T11:05:50.518 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster_var_name=local_cluster 2023-10-11T11:05:50.518 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance_var_name=instance 2023-10-11T11:05:50.518 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:05:50.518 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=0 2023-10-11T11:05:50.518 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ '[' 0 = cluster1:0 ']' 2023-10-11T11:05:50.519 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval local_cluster=cluster1 2023-10-11T11:05:50.519 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ local_cluster=cluster1 2023-10-11T11:05:50.519 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval instance=0 2023-10-11T11:05:50.519 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ instance=0 2023-10-11T11:05:50.519 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-10-11T11:05:50.532 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T11:05:50.532 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T11:05:50.533 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T11:05:50.533 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_24 2023-10-11T11:05:50.633 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:Invalid command: unused arguments: ['mirror/image_24'] 2023-10-11T11:05:50.634 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd mirror status : get status for rbd mirror 2023-10-11T11:05:50.634 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:admin_socket: invalid command 2023-10-11T11:05:50.637 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-10-11T11:05:50.637 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_24 2023-10-11T11:05:50.637 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1:1 2023-10-11T11:05:50.638 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T11:05:50.638 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance 2023-10-11T11:05:50.638 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-10-11T11:05:50.638 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local val=cluster1:1 2023-10-11T11:05:50.639 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_var_name=cluster 2023-10-11T11:05:50.639 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance_var_name=instance 2023-10-11T11:05:50.639 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ cluster=cluster1 2023-10-11T11:05:50.639 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ instance=1 2023-10-11T11:05:50.639 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' 1 = cluster1:1 ']' 2023-10-11T11:05:50.640 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval cluster=cluster1 2023-10-11T11:05:50.640 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:05:50.640 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval instance=1 2023-10-11T11:05:50.640 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=1 2023-10-11T11:05:50.641 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-10-11T11:05:50.641 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local local_cluster=cluster1:1 2023-10-11T11:05:50.641 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster=cluster1 2023-10-11T11:05:50.641 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance 2023-10-11T11:05:50.641 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-10-11T11:05:50.642 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local val=cluster1:1 2023-10-11T11:05:50.642 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster_var_name=local_cluster 2023-10-11T11:05:50.642 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance_var_name=instance 2023-10-11T11:05:50.642 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:05:50.642 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=1 2023-10-11T11:05:50.643 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ '[' 1 = cluster1:1 ']' 2023-10-11T11:05:50.643 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval local_cluster=cluster1 2023-10-11T11:05:50.643 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ local_cluster=cluster1 2023-10-11T11:05:50.643 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval instance=1 2023-10-11T11:05:50.643 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ instance=1 2023-10-11T11:05:50.644 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-10-11T11:05:50.657 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T11:05:50.657 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T11:05:50.657 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T11:05:50.657 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_24 2023-10-11T11:05:50.807 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ return 0 2023-10-11T11:05:50.808 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ status_result=' "state": "Replaying"' 2023-10-11T11:05:50.808 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ echo ' "state": "Replaying"' 2023-10-11T11:05:50.808 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ grep -i Replaying 2023-10-11T11:05:50.809 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: "state": "Replaying" 2023-10-11T11:05:50.810 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ current_state=started 2023-10-11T11:05:50.811 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test started = started 2023-10-11T11:05:50.811 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:05:50.811 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_24 2023-10-11T11:05:50.811 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local local_cluster=cluster1 2023-10-11T11:05:50.811 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:05:50.812 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:05:50.812 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_24 2023-10-11T11:05:50.812 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' snapshot = journal ']' 2023-10-11T11:05:50.812 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' snapshot = snapshot ']' 2023-10-11T11:05:50.812 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_24 2023-10-11T11:05:50.813 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local local_cluster=cluster1 2023-10-11T11:05:50.813 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:05:50.813 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:05:50.813 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_24 2023-10-11T11:05:50.813 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ mkfname cluster2-mirror-image_24.status 2023-10-11T11:05:50.813 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ echo cluster2-mirror-image_24.status 2023-10-11T11:05:50.814 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -e 's|[/ ]|_|g' 2023-10-11T11:05:50.814 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local status_log=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_24.status 2023-10-11T11:05:50.815 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ mkfname cluster1-mirror-image_24.status 2023-10-11T11:05:50.815 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ echo cluster1-mirror-image_24.status 2023-10-11T11:05:50.815 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -e 's|[/ ]|_|g' 2023-10-11T11:05:50.815 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local local_status_log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_24.status 2023-10-11T11:05:50.816 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ mirror_image_snapshot cluster2 mirror image_24 2023-10-11T11:05:50.816 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:05:50.816 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:05:50.816 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_24 2023-10-11T11:05:50.816 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_24 2023-10-11T11:05:51.502 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:Snapshot ID: 165 2023-10-11T11:05:51.514 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_24 /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_24.status 2023-10-11T11:05:51.514 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:05:51.514 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:05:51.514 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_24 2023-10-11T11:05:51.515 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_24.status 2023-10-11T11:05:51.515 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_24 --format xml 2023-10-11T11:05:51.515 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:05:51.556 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-10-11T11:05:51.558 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snapshot_id=165 2023-10-11T11:05:51.558 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T11:05:51.558 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T11:05:51.558 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.2 2023-10-11T11:05:51.759 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_24 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_24.status 2023-10-11T11:05:51.759 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:05:51.760 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:05:51.760 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_24 2023-10-11T11:05:51.760 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_24.status 2023-10-11T11:05:51.760 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_24 --format xml 2023-10-11T11:05:51.760 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:05:51.805 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T11:05:51.805 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T11:05:51.807 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:-:1.1: Document is empty 2023-10-11T11:05:51.807 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: 2023-10-11T11:05:51.807 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:^ 2023-10-11T11:05:51.807 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id= 2023-10-11T11:05:51.808 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 165 = '' 2023-10-11T11:05:51.808 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T11:05:51.808 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.4 2023-10-11T11:05:52.208 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_24 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_24.status 2023-10-11T11:05:52.208 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:05:52.208 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:05:52.209 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_24 2023-10-11T11:05:52.209 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_24.status 2023-10-11T11:05:52.209 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_24 --format xml 2023-10-11T11:05:52.209 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:05:52.256 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T11:05:52.256 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T11:05:52.257 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:-:1.1: Document is empty 2023-10-11T11:05:52.257 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: 2023-10-11T11:05:52.258 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:^ 2023-10-11T11:05:52.258 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id= 2023-10-11T11:05:52.258 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 165 = '' 2023-10-11T11:05:52.258 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T11:05:52.259 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.8 2023-10-11T11:05:53.059 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_24 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_24.status 2023-10-11T11:05:53.059 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:05:53.060 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:05:53.060 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_24 2023-10-11T11:05:53.060 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_24.status 2023-10-11T11:05:53.060 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_24 --format xml 2023-10-11T11:05:53.060 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:05:53.103 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T11:05:53.105 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id=165 2023-10-11T11:05:53.105 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 165 = 165 2023-10-11T11:05:53.106 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:05:53.106 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_snap_present cluster1 mirror image_24 snap 2023-10-11T11:05:53.106 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:05:53.106 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:05:53.106 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_24 2023-10-11T11:05:53.107 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap 2023-10-11T11:05:53.107 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T11:05:53.107 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-10-11T11:05:53.107 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 1 2023-10-11T11:05:53.637 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.986064677056822) > weight (0.25) 2023-10-11T11:05:53.637 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.3564888871522297) > weight (0.25) 2023-10-11T11:05:53.638 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.8733360916018744) > weight (0.25) 2023-10-11T11:05:53.638 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.40769962895278633) > weight (0.25) 2023-10-11T11:05:53.638 INFO:tasks.rbd_mirror_thrash:waiting for 62 secs before thrashing 2023-10-11T11:05:54.106 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 info mirror/image_24@snap 2023-10-11T11:05:54.152 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:rbd image 'image_24': 2023-10-11T11:05:54.152 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: size 128 MiB in 32 objects 2023-10-11T11:05:54.152 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: order 22 (4 MiB objects) 2023-10-11T11:05:54.153 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: snapshot_count: 2 2023-10-11T11:05:54.153 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: id: 1289a94eb24d 2023-10-11T11:05:54.153 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: block_name_prefix: rbd_data.1289a94eb24d 2023-10-11T11:05:54.153 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: format: 2 2023-10-11T11:05:54.154 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-10-11T11:05:54.154 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: op_features: 2023-10-11T11:05:54.154 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: flags: 2023-10-11T11:05:54.154 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: create_timestamp: Wed Oct 11 10:58:07 2023 2023-10-11T11:05:54.155 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: access_timestamp: Wed Oct 11 10:58:07 2023 2023-10-11T11:05:54.155 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: modify_timestamp: Wed Oct 11 10:58:07 2023 2023-10-11T11:05:54.155 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: protected: False 2023-10-11T11:05:54.156 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring state: enabled 2023-10-11T11:05:54.156 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring mode: snapshot 2023-10-11T11:05:54.156 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring global id: 1d287991-2520-4320-9427-8214deba681d 2023-10-11T11:05:54.156 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring primary: false 2023-10-11T11:05:54.158 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:05:54.158 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ compare_image_snaps mirror image_24 snap 2023-10-11T11:05:54.159 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:05:54.159 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_24 2023-10-11T11:05:54.159 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap 2023-10-11T11:05:54.159 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local ret=0 2023-10-11T11:05:54.160 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local rmt_export=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_24.export 2023-10-11T11:05:54.160 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local loc_export=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_24.export 2023-10-11T11:05:54.160 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rm -f /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_24.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_24.export 2023-10-11T11:05:54.160 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 -p mirror export image_24@snap /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_24.export 2023-10-11T11:05:54.335 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-10-11T11:05:54.341 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 -p mirror export image_24@snap /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_24.export 2023-10-11T11:05:54.578 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-10-11T11:05:54.587 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ cmp /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_24.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_24.export 2023-10-11T11:05:54.638 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rm -f /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_24.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_24.export 2023-10-11T11:05:54.675 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:05:54.675 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T11:05:54.675 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_25 2023-10-11T11:05:54.676 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ create_snap cluster2 mirror image_25 snap 2023-10-11T11:05:54.676 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:05:54.676 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:05:54.676 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_25 2023-10-11T11:05:54.676 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap 2023-10-11T11:05:54.677 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 -p mirror snap create image_25@snap --debug-rbd=20 --debug-journaler=20 2023-10-11T11:05:56.525 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_replay_started cluster1 mirror image_25 2023-10-11T11:05:56.526 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:05:56.526 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:05:56.526 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_25 2023-10-11T11:05:56.526 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_replay_state cluster1 mirror image_25 started 2023-10-11T11:05:56.526 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:05:56.527 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:05:56.527 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_25 2023-10-11T11:05:56.527 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local state=started 2023-10-11T11:05:56.527 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T11:05:56.527 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-10-11T11:05:56.527 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 1 2023-10-11T11:05:57.526 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test_image_replay_state cluster1 mirror image_25 started 2023-10-11T11:05:57.527 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:05:57.527 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:05:57.527 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_25 2023-10-11T11:05:57.527 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local test_state=started 2023-10-11T11:05:57.527 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local status_result 2023-10-11T11:05:57.528 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_state=stopped 2023-10-11T11:05:57.529 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_25 2023-10-11T11:05:57.529 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_instance=cluster1 2023-10-11T11:05:57.529 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T11:05:57.529 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ grep -i state 2023-10-11T11:05:57.530 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:05:57.530 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance=cluster1 2023-10-11T11:05:57.530 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local loop_instance 2023-10-11T11:05:57.530 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-10-11T11:05:57.530 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sleep 0 2023-10-11T11:05:57.530 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-10-11T11:05:57.531 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ seq 0 3 2023-10-11T11:05:57.532 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-10-11T11:05:57.532 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_25 2023-10-11T11:05:57.532 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1:0 2023-10-11T11:05:57.533 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T11:05:57.533 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance 2023-10-11T11:05:57.533 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-10-11T11:05:57.533 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local val=cluster1:0 2023-10-11T11:05:57.533 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_var_name=cluster 2023-10-11T11:05:57.533 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance_var_name=instance 2023-10-11T11:05:57.534 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ cluster=cluster1 2023-10-11T11:05:57.534 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ instance=0 2023-10-11T11:05:57.534 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' 0 = cluster1:0 ']' 2023-10-11T11:05:57.534 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval cluster=cluster1 2023-10-11T11:05:57.534 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:05:57.534 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval instance=0 2023-10-11T11:05:57.535 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=0 2023-10-11T11:05:57.536 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-10-11T11:05:57.536 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local local_cluster=cluster1:0 2023-10-11T11:05:57.536 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster=cluster1 2023-10-11T11:05:57.537 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance 2023-10-11T11:05:57.537 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-10-11T11:05:57.537 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local val=cluster1:0 2023-10-11T11:05:57.537 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster_var_name=local_cluster 2023-10-11T11:05:57.537 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance_var_name=instance 2023-10-11T11:05:57.538 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:05:57.538 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=0 2023-10-11T11:05:57.538 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ '[' 0 = cluster1:0 ']' 2023-10-11T11:05:57.538 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval local_cluster=cluster1 2023-10-11T11:05:57.538 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ local_cluster=cluster1 2023-10-11T11:05:57.538 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval instance=0 2023-10-11T11:05:57.539 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ instance=0 2023-10-11T11:05:57.539 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-10-11T11:05:57.555 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T11:05:57.556 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T11:05:57.556 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T11:05:57.556 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_25 2023-10-11T11:05:57.655 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:Invalid command: unused arguments: ['mirror/image_25'] 2023-10-11T11:05:57.655 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd mirror status : get status for rbd mirror 2023-10-11T11:05:57.655 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:admin_socket: invalid command 2023-10-11T11:05:57.658 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-10-11T11:05:57.658 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_25 2023-10-11T11:05:57.658 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1:1 2023-10-11T11:05:57.659 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T11:05:57.659 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance 2023-10-11T11:05:57.659 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-10-11T11:05:57.659 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local val=cluster1:1 2023-10-11T11:05:57.659 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_var_name=cluster 2023-10-11T11:05:57.659 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance_var_name=instance 2023-10-11T11:05:57.660 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ cluster=cluster1 2023-10-11T11:05:57.660 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ instance=1 2023-10-11T11:05:57.660 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' 1 = cluster1:1 ']' 2023-10-11T11:05:57.660 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval cluster=cluster1 2023-10-11T11:05:57.660 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:05:57.660 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval instance=1 2023-10-11T11:05:57.661 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=1 2023-10-11T11:05:57.661 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-10-11T11:05:57.661 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local local_cluster=cluster1:1 2023-10-11T11:05:57.661 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster=cluster1 2023-10-11T11:05:57.661 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance 2023-10-11T11:05:57.661 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-10-11T11:05:57.662 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local val=cluster1:1 2023-10-11T11:05:57.662 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster_var_name=local_cluster 2023-10-11T11:05:57.662 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance_var_name=instance 2023-10-11T11:05:57.662 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:05:57.662 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=1 2023-10-11T11:05:57.662 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ '[' 1 = cluster1:1 ']' 2023-10-11T11:05:57.663 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval local_cluster=cluster1 2023-10-11T11:05:57.663 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ local_cluster=cluster1 2023-10-11T11:05:57.663 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval instance=1 2023-10-11T11:05:57.663 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ instance=1 2023-10-11T11:05:57.663 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-10-11T11:05:57.678 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T11:05:57.678 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T11:05:57.678 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T11:05:57.678 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_25 2023-10-11T11:05:57.826 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ return 0 2023-10-11T11:05:57.826 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ status_result=' "state": "Replaying"' 2023-10-11T11:05:57.826 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ echo ' "state": "Replaying"' 2023-10-11T11:05:57.827 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ grep -i Replaying 2023-10-11T11:05:57.827 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: "state": "Replaying" 2023-10-11T11:05:57.828 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ current_state=started 2023-10-11T11:05:57.829 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test started = started 2023-10-11T11:05:57.829 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:05:57.829 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_25 2023-10-11T11:05:57.829 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local local_cluster=cluster1 2023-10-11T11:05:57.829 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:05:57.830 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:05:57.830 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_25 2023-10-11T11:05:57.830 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' snapshot = journal ']' 2023-10-11T11:05:57.830 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' snapshot = snapshot ']' 2023-10-11T11:05:57.830 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_25 2023-10-11T11:05:57.830 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local local_cluster=cluster1 2023-10-11T11:05:57.831 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:05:57.831 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:05:57.831 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_25 2023-10-11T11:05:57.831 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ mkfname cluster2-mirror-image_25.status 2023-10-11T11:05:57.831 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ echo cluster2-mirror-image_25.status 2023-10-11T11:05:57.831 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -e 's|[/ ]|_|g' 2023-10-11T11:05:57.832 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local status_log=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_25.status 2023-10-11T11:05:57.833 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ mkfname cluster1-mirror-image_25.status 2023-10-11T11:05:57.833 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ echo cluster1-mirror-image_25.status 2023-10-11T11:05:57.833 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -e 's|[/ ]|_|g' 2023-10-11T11:05:57.833 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local local_status_log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_25.status 2023-10-11T11:05:57.833 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ mirror_image_snapshot cluster2 mirror image_25 2023-10-11T11:05:57.834 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:05:57.834 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:05:57.834 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_25 2023-10-11T11:05:57.834 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_25 2023-10-11T11:05:58.522 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:Snapshot ID: 167 2023-10-11T11:05:58.533 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_25 /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_25.status 2023-10-11T11:05:58.533 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:05:58.534 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:05:58.534 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_25 2023-10-11T11:05:58.534 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_25.status 2023-10-11T11:05:58.535 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_25 --format xml 2023-10-11T11:05:58.535 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:05:58.576 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-10-11T11:05:58.578 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snapshot_id=167 2023-10-11T11:05:58.579 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T11:05:58.579 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T11:05:58.579 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.2 2023-10-11T11:05:58.780 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_25 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_25.status 2023-10-11T11:05:58.780 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:05:58.780 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:05:58.781 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_25 2023-10-11T11:05:58.781 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_25.status 2023-10-11T11:05:58.781 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_25 --format xml 2023-10-11T11:05:58.781 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:05:58.826 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T11:05:58.826 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T11:05:58.828 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:-:1.1: Document is empty 2023-10-11T11:05:58.828 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: 2023-10-11T11:05:58.828 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:^ 2023-10-11T11:05:58.829 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id= 2023-10-11T11:05:58.829 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 167 = '' 2023-10-11T11:05:58.829 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T11:05:58.829 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.4 2023-10-11T11:05:59.229 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_25 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_25.status 2023-10-11T11:05:59.229 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:05:59.230 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:05:59.230 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_25 2023-10-11T11:05:59.230 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_25.status 2023-10-11T11:05:59.231 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_25 --format xml 2023-10-11T11:05:59.231 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:05:59.274 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T11:05:59.274 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T11:05:59.276 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:-:1.1: Document is empty 2023-10-11T11:05:59.276 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: 2023-10-11T11:05:59.276 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:^ 2023-10-11T11:05:59.277 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id= 2023-10-11T11:05:59.277 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 167 = '' 2023-10-11T11:05:59.277 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T11:05:59.277 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.8 2023-10-11T11:06:00.077 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_25 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_25.status 2023-10-11T11:06:00.077 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:06:00.078 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:06:00.078 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_25 2023-10-11T11:06:00.078 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_25.status 2023-10-11T11:06:00.078 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_25 --format xml 2023-10-11T11:06:00.078 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:06:00.121 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T11:06:00.123 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id=167 2023-10-11T11:06:00.124 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 167 = 167 2023-10-11T11:06:00.124 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:06:00.124 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_snap_present cluster1 mirror image_25 snap 2023-10-11T11:06:00.124 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:06:00.124 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:06:00.125 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_25 2023-10-11T11:06:00.125 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap 2023-10-11T11:06:00.125 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T11:06:00.125 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-10-11T11:06:00.125 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 1 2023-10-11T11:06:01.124 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 info mirror/image_25@snap 2023-10-11T11:06:01.169 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:rbd image 'image_25': 2023-10-11T11:06:01.170 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: size 128 MiB in 32 objects 2023-10-11T11:06:01.170 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: order 22 (4 MiB objects) 2023-10-11T11:06:01.170 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: snapshot_count: 2 2023-10-11T11:06:01.170 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: id: 12892f81f9e9 2023-10-11T11:06:01.171 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: block_name_prefix: rbd_data.12892f81f9e9 2023-10-11T11:06:01.171 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: format: 2 2023-10-11T11:06:01.171 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-10-11T11:06:01.171 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: op_features: 2023-10-11T11:06:01.172 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: flags: 2023-10-11T11:06:01.172 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: create_timestamp: Wed Oct 11 10:58:16 2023 2023-10-11T11:06:01.172 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: access_timestamp: Wed Oct 11 10:58:16 2023 2023-10-11T11:06:01.172 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: modify_timestamp: Wed Oct 11 10:58:16 2023 2023-10-11T11:06:01.173 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: protected: False 2023-10-11T11:06:01.173 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring state: enabled 2023-10-11T11:06:01.173 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring mode: snapshot 2023-10-11T11:06:01.173 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring global id: 48104021-9df0-46cf-a4dc-ab4fb67e4fe1 2023-10-11T11:06:01.173 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring primary: false 2023-10-11T11:06:01.175 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:06:01.175 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ compare_image_snaps mirror image_25 snap 2023-10-11T11:06:01.175 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:06:01.175 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_25 2023-10-11T11:06:01.176 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap 2023-10-11T11:06:01.176 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local ret=0 2023-10-11T11:06:01.176 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local rmt_export=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_25.export 2023-10-11T11:06:01.176 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local loc_export=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_25.export 2023-10-11T11:06:01.176 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rm -f /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_25.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_25.export 2023-10-11T11:06:01.177 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 -p mirror export image_25@snap /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_25.export 2023-10-11T11:06:01.344 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-10-11T11:06:01.349 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 -p mirror export image_25@snap /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_25.export 2023-10-11T11:06:01.554 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-10-11T11:06:01.559 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ cmp /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_25.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_25.export 2023-10-11T11:06:01.613 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rm -f /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_25.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_25.export 2023-10-11T11:06:01.650 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:06:01.650 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T11:06:01.650 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_26 2023-10-11T11:06:01.650 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ create_snap cluster2 mirror image_26 snap 2023-10-11T11:06:01.650 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:06:01.651 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:06:01.651 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_26 2023-10-11T11:06:01.651 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap 2023-10-11T11:06:01.651 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 -p mirror snap create image_26@snap --debug-rbd=20 --debug-journaler=20 2023-10-11T11:06:03.537 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_replay_started cluster1 mirror image_26 2023-10-11T11:06:03.537 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:06:03.538 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:06:03.538 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_26 2023-10-11T11:06:03.538 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_replay_state cluster1 mirror image_26 started 2023-10-11T11:06:03.538 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:06:03.539 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:06:03.539 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_26 2023-10-11T11:06:03.539 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local state=started 2023-10-11T11:06:03.540 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T11:06:03.540 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-10-11T11:06:03.540 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 1 2023-10-11T11:06:04.539 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test_image_replay_state cluster1 mirror image_26 started 2023-10-11T11:06:04.539 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:06:04.540 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:06:04.540 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_26 2023-10-11T11:06:04.540 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local test_state=started 2023-10-11T11:06:04.540 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local status_result 2023-10-11T11:06:04.541 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_state=stopped 2023-10-11T11:06:04.541 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_26 2023-10-11T11:06:04.541 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_instance=cluster1 2023-10-11T11:06:04.542 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T11:06:04.542 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ grep -i state 2023-10-11T11:06:04.542 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:06:04.542 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance=cluster1 2023-10-11T11:06:04.543 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local loop_instance 2023-10-11T11:06:04.543 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-10-11T11:06:04.543 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sleep 0 2023-10-11T11:06:04.547 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-10-11T11:06:04.547 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ seq 0 3 2023-10-11T11:06:04.548 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-10-11T11:06:04.548 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_26 2023-10-11T11:06:04.548 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1:0 2023-10-11T11:06:04.548 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T11:06:04.549 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance 2023-10-11T11:06:04.549 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-10-11T11:06:04.549 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local val=cluster1:0 2023-10-11T11:06:04.550 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_var_name=cluster 2023-10-11T11:06:04.550 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance_var_name=instance 2023-10-11T11:06:04.550 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ cluster=cluster1 2023-10-11T11:06:04.550 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ instance=0 2023-10-11T11:06:04.551 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' 0 = cluster1:0 ']' 2023-10-11T11:06:04.551 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval cluster=cluster1 2023-10-11T11:06:04.551 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:06:04.551 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval instance=0 2023-10-11T11:06:04.552 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=0 2023-10-11T11:06:04.552 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-10-11T11:06:04.552 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local local_cluster=cluster1:0 2023-10-11T11:06:04.552 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster=cluster1 2023-10-11T11:06:04.553 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance 2023-10-11T11:06:04.553 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-10-11T11:06:04.553 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local val=cluster1:0 2023-10-11T11:06:04.553 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster_var_name=local_cluster 2023-10-11T11:06:04.554 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance_var_name=instance 2023-10-11T11:06:04.554 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:06:04.554 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=0 2023-10-11T11:06:04.554 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ '[' 0 = cluster1:0 ']' 2023-10-11T11:06:04.555 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval local_cluster=cluster1 2023-10-11T11:06:04.555 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ local_cluster=cluster1 2023-10-11T11:06:04.555 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval instance=0 2023-10-11T11:06:04.555 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ instance=0 2023-10-11T11:06:04.556 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-10-11T11:06:04.565 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T11:06:04.565 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T11:06:04.565 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T11:06:04.566 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_26 2023-10-11T11:06:04.667 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:Invalid command: unused arguments: ['mirror/image_26'] 2023-10-11T11:06:04.667 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd mirror status : get status for rbd mirror 2023-10-11T11:06:04.667 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:admin_socket: invalid command 2023-10-11T11:06:04.671 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-10-11T11:06:04.671 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_26 2023-10-11T11:06:04.672 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1:1 2023-10-11T11:06:04.672 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T11:06:04.672 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance 2023-10-11T11:06:04.673 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-10-11T11:06:04.673 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local val=cluster1:1 2023-10-11T11:06:04.673 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_var_name=cluster 2023-10-11T11:06:04.673 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance_var_name=instance 2023-10-11T11:06:04.674 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ cluster=cluster1 2023-10-11T11:06:04.674 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ instance=1 2023-10-11T11:06:04.674 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' 1 = cluster1:1 ']' 2023-10-11T11:06:04.674 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval cluster=cluster1 2023-10-11T11:06:04.675 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:06:04.675 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval instance=1 2023-10-11T11:06:04.675 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=1 2023-10-11T11:06:04.675 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-10-11T11:06:04.676 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local local_cluster=cluster1:1 2023-10-11T11:06:04.676 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster=cluster1 2023-10-11T11:06:04.676 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance 2023-10-11T11:06:04.677 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-10-11T11:06:04.677 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local val=cluster1:1 2023-10-11T11:06:04.677 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster_var_name=local_cluster 2023-10-11T11:06:04.677 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance_var_name=instance 2023-10-11T11:06:04.678 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:06:04.678 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=1 2023-10-11T11:06:04.678 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ '[' 1 = cluster1:1 ']' 2023-10-11T11:06:04.678 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval local_cluster=cluster1 2023-10-11T11:06:04.679 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ local_cluster=cluster1 2023-10-11T11:06:04.679 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval instance=1 2023-10-11T11:06:04.679 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ instance=1 2023-10-11T11:06:04.679 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-10-11T11:06:04.689 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T11:06:04.689 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T11:06:04.690 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T11:06:04.690 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_26 2023-10-11T11:06:04.841 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ return 0 2023-10-11T11:06:04.842 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ status_result=' "state": "Replaying"' 2023-10-11T11:06:04.842 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ echo ' "state": "Replaying"' 2023-10-11T11:06:04.842 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ grep -i Replaying 2023-10-11T11:06:04.843 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: "state": "Replaying" 2023-10-11T11:06:04.846 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ current_state=started 2023-10-11T11:06:04.847 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test started = started 2023-10-11T11:06:04.847 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:06:04.847 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_26 2023-10-11T11:06:04.847 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local local_cluster=cluster1 2023-10-11T11:06:04.848 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:06:04.848 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:06:04.848 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_26 2023-10-11T11:06:04.848 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' snapshot = journal ']' 2023-10-11T11:06:04.849 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' snapshot = snapshot ']' 2023-10-11T11:06:04.849 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_26 2023-10-11T11:06:04.849 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local local_cluster=cluster1 2023-10-11T11:06:04.850 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:06:04.850 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:06:04.850 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_26 2023-10-11T11:06:04.850 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ mkfname cluster2-mirror-image_26.status 2023-10-11T11:06:04.851 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ echo cluster2-mirror-image_26.status 2023-10-11T11:06:04.851 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -e 's|[/ ]|_|g' 2023-10-11T11:06:04.851 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local status_log=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_26.status 2023-10-11T11:06:04.851 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ mkfname cluster1-mirror-image_26.status 2023-10-11T11:06:04.852 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ echo cluster1-mirror-image_26.status 2023-10-11T11:06:04.852 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -e 's|[/ ]|_|g' 2023-10-11T11:06:04.852 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local local_status_log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_26.status 2023-10-11T11:06:04.853 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ mirror_image_snapshot cluster2 mirror image_26 2023-10-11T11:06:04.853 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:06:04.853 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:06:04.853 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_26 2023-10-11T11:06:04.854 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_26 2023-10-11T11:06:05.541 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:Snapshot ID: 169 2023-10-11T11:06:05.553 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_26 /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_26.status 2023-10-11T11:06:05.553 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:06:05.554 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:06:05.554 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_26 2023-10-11T11:06:05.554 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_26.status 2023-10-11T11:06:05.554 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_26 --format xml 2023-10-11T11:06:05.555 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:06:05.596 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-10-11T11:06:05.599 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snapshot_id=169 2023-10-11T11:06:05.599 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T11:06:05.599 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T11:06:05.599 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.2 2023-10-11T11:06:05.799 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_26 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_26.status 2023-10-11T11:06:05.799 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:06:05.800 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:06:05.800 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_26 2023-10-11T11:06:05.800 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_26.status 2023-10-11T11:06:05.800 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_26 --format xml 2023-10-11T11:06:05.800 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:06:05.846 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T11:06:05.848 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id=93 2023-10-11T11:06:05.848 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 169 = 93 2023-10-11T11:06:05.848 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T11:06:05.849 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.4 2023-10-11T11:06:06.250 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_26 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_26.status 2023-10-11T11:06:06.250 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:06:06.250 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:06:06.251 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_26 2023-10-11T11:06:06.251 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_26.status 2023-10-11T11:06:06.251 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_26 --format xml 2023-10-11T11:06:06.251 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:06:06.299 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T11:06:06.299 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T11:06:06.301 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:-:1.1: Document is empty 2023-10-11T11:06:06.301 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: 2023-10-11T11:06:06.301 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:^ 2023-10-11T11:06:06.301 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id= 2023-10-11T11:06:06.302 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 169 = '' 2023-10-11T11:06:06.302 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T11:06:06.302 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.8 2023-10-11T11:06:07.102 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_26 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_26.status 2023-10-11T11:06:07.103 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:06:07.103 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:06:07.103 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_26 2023-10-11T11:06:07.104 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_26.status 2023-10-11T11:06:07.104 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_26 --format xml 2023-10-11T11:06:07.104 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:06:07.193 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T11:06:07.194 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T11:06:07.195 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:-:1.1: Document is empty 2023-10-11T11:06:07.195 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: 2023-10-11T11:06:07.196 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:^ 2023-10-11T11:06:07.196 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id= 2023-10-11T11:06:07.196 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 169 = '' 2023-10-11T11:06:07.196 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T11:06:07.197 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 1.6 2023-10-11T11:06:08.797 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_26 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_26.status 2023-10-11T11:06:08.797 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:06:08.798 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:06:08.798 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_26 2023-10-11T11:06:08.798 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_26.status 2023-10-11T11:06:08.799 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_26 --format xml 2023-10-11T11:06:08.799 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:06:08.847 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T11:06:08.850 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id=169 2023-10-11T11:06:08.850 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 169 = 169 2023-10-11T11:06:08.850 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:06:08.851 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_snap_present cluster1 mirror image_26 snap 2023-10-11T11:06:08.851 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:06:08.851 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:06:08.851 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_26 2023-10-11T11:06:08.852 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap 2023-10-11T11:06:08.852 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T11:06:08.852 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-10-11T11:06:08.852 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 1 2023-10-11T11:06:09.850 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 info mirror/image_26@snap 2023-10-11T11:06:09.896 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:rbd image 'image_26': 2023-10-11T11:06:09.896 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: size 128 MiB in 32 objects 2023-10-11T11:06:09.897 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: order 22 (4 MiB objects) 2023-10-11T11:06:09.897 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: snapshot_count: 2 2023-10-11T11:06:09.897 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: id: 12891110160b 2023-10-11T11:06:09.897 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: block_name_prefix: rbd_data.12891110160b 2023-10-11T11:06:09.897 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: format: 2 2023-10-11T11:06:09.897 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-10-11T11:06:09.898 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: op_features: 2023-10-11T11:06:09.898 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: flags: 2023-10-11T11:06:09.898 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: create_timestamp: Wed Oct 11 10:58:26 2023 2023-10-11T11:06:09.898 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: access_timestamp: Wed Oct 11 10:58:26 2023 2023-10-11T11:06:09.898 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: modify_timestamp: Wed Oct 11 10:58:26 2023 2023-10-11T11:06:09.899 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: protected: False 2023-10-11T11:06:09.899 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring state: enabled 2023-10-11T11:06:09.899 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring mode: snapshot 2023-10-11T11:06:09.899 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring global id: 76bafcc4-4e59-4baf-875a-22631193e31e 2023-10-11T11:06:09.899 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring primary: false 2023-10-11T11:06:09.900 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:06:09.900 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ compare_image_snaps mirror image_26 snap 2023-10-11T11:06:09.901 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:06:09.901 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_26 2023-10-11T11:06:09.901 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap 2023-10-11T11:06:09.901 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local ret=0 2023-10-11T11:06:09.901 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local rmt_export=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_26.export 2023-10-11T11:06:09.902 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local loc_export=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_26.export 2023-10-11T11:06:09.902 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rm -f /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_26.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_26.export 2023-10-11T11:06:09.902 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 -p mirror export image_26@snap /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_26.export 2023-10-11T11:06:10.076 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-10-11T11:06:10.081 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 -p mirror export image_26@snap /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_26.export 2023-10-11T11:06:10.340 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-10-11T11:06:10.347 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ cmp /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_26.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_26.export 2023-10-11T11:06:10.403 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rm -f /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_26.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_26.export 2023-10-11T11:06:10.439 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:06:10.440 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T11:06:10.440 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_27 2023-10-11T11:06:10.440 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ create_snap cluster2 mirror image_27 snap 2023-10-11T11:06:10.440 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:06:10.440 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:06:10.441 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_27 2023-10-11T11:06:10.441 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap 2023-10-11T11:06:10.441 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 -p mirror snap create image_27@snap --debug-rbd=20 --debug-journaler=20 2023-10-11T11:06:12.410 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_replay_started cluster1 mirror image_27 2023-10-11T11:06:12.411 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:06:12.411 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:06:12.411 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_27 2023-10-11T11:06:12.411 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_replay_state cluster1 mirror image_27 started 2023-10-11T11:06:12.412 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:06:12.412 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:06:12.412 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_27 2023-10-11T11:06:12.412 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local state=started 2023-10-11T11:06:12.412 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T11:06:12.413 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-10-11T11:06:12.413 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 1 2023-10-11T11:06:13.412 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test_image_replay_state cluster1 mirror image_27 started 2023-10-11T11:06:13.412 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:06:13.412 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:06:13.413 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_27 2023-10-11T11:06:13.413 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local test_state=started 2023-10-11T11:06:13.413 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local status_result 2023-10-11T11:06:13.413 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_state=stopped 2023-10-11T11:06:13.417 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_27 2023-10-11T11:06:13.417 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ grep -i state 2023-10-11T11:06:13.417 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_instance=cluster1 2023-10-11T11:06:13.417 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T11:06:13.418 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:06:13.418 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance=cluster1 2023-10-11T11:06:13.418 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local loop_instance 2023-10-11T11:06:13.419 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-10-11T11:06:13.419 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sleep 0 2023-10-11T11:06:13.419 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-10-11T11:06:13.419 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ seq 0 3 2023-10-11T11:06:13.420 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-10-11T11:06:13.420 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_27 2023-10-11T11:06:13.420 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1:0 2023-10-11T11:06:13.420 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T11:06:13.421 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance 2023-10-11T11:06:13.421 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-10-11T11:06:13.421 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local val=cluster1:0 2023-10-11T11:06:13.421 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_var_name=cluster 2023-10-11T11:06:13.422 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance_var_name=instance 2023-10-11T11:06:13.422 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ cluster=cluster1 2023-10-11T11:06:13.422 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ instance=0 2023-10-11T11:06:13.422 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' 0 = cluster1:0 ']' 2023-10-11T11:06:13.423 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval cluster=cluster1 2023-10-11T11:06:13.423 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:06:13.423 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval instance=0 2023-10-11T11:06:13.423 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=0 2023-10-11T11:06:13.424 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-10-11T11:06:13.424 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local local_cluster=cluster1:0 2023-10-11T11:06:13.424 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster=cluster1 2023-10-11T11:06:13.424 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance 2023-10-11T11:06:13.425 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-10-11T11:06:13.425 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local val=cluster1:0 2023-10-11T11:06:13.426 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster_var_name=local_cluster 2023-10-11T11:06:13.427 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance_var_name=instance 2023-10-11T11:06:13.427 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:06:13.427 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=0 2023-10-11T11:06:13.427 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ '[' 0 = cluster1:0 ']' 2023-10-11T11:06:13.428 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval local_cluster=cluster1 2023-10-11T11:06:13.428 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ local_cluster=cluster1 2023-10-11T11:06:13.428 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval instance=0 2023-10-11T11:06:13.428 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ instance=0 2023-10-11T11:06:13.429 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-10-11T11:06:13.439 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T11:06:13.439 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T11:06:13.440 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T11:06:13.440 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_27 2023-10-11T11:06:13.541 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:Invalid command: unused arguments: ['mirror/image_27'] 2023-10-11T11:06:13.541 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd mirror status : get status for rbd mirror 2023-10-11T11:06:13.541 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:admin_socket: invalid command 2023-10-11T11:06:13.544 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-10-11T11:06:13.545 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_27 2023-10-11T11:06:13.545 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1:1 2023-10-11T11:06:13.545 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T11:06:13.546 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance 2023-10-11T11:06:13.546 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-10-11T11:06:13.546 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local val=cluster1:1 2023-10-11T11:06:13.546 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_var_name=cluster 2023-10-11T11:06:13.547 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance_var_name=instance 2023-10-11T11:06:13.547 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ cluster=cluster1 2023-10-11T11:06:13.547 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ instance=1 2023-10-11T11:06:13.547 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' 1 = cluster1:1 ']' 2023-10-11T11:06:13.548 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval cluster=cluster1 2023-10-11T11:06:13.548 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:06:13.548 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval instance=1 2023-10-11T11:06:13.548 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=1 2023-10-11T11:06:13.549 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-10-11T11:06:13.549 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local local_cluster=cluster1:1 2023-10-11T11:06:13.549 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster=cluster1 2023-10-11T11:06:13.549 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance 2023-10-11T11:06:13.550 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-10-11T11:06:13.550 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local val=cluster1:1 2023-10-11T11:06:13.550 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster_var_name=local_cluster 2023-10-11T11:06:13.550 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance_var_name=instance 2023-10-11T11:06:13.551 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:06:13.551 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=1 2023-10-11T11:06:13.551 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ '[' 1 = cluster1:1 ']' 2023-10-11T11:06:13.551 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval local_cluster=cluster1 2023-10-11T11:06:13.552 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ local_cluster=cluster1 2023-10-11T11:06:13.552 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval instance=1 2023-10-11T11:06:13.552 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ instance=1 2023-10-11T11:06:13.552 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-10-11T11:06:13.562 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T11:06:13.563 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T11:06:13.563 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T11:06:13.563 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_27 2023-10-11T11:06:13.710 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ return 0 2023-10-11T11:06:13.711 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ status_result=' "state": "Replaying"' 2023-10-11T11:06:13.711 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ echo ' "state": "Replaying"' 2023-10-11T11:06:13.711 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ grep -i Replaying 2023-10-11T11:06:13.712 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: "state": "Replaying" 2023-10-11T11:06:13.714 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ current_state=started 2023-10-11T11:06:13.715 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test started = started 2023-10-11T11:06:13.715 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:06:13.715 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_27 2023-10-11T11:06:13.715 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local local_cluster=cluster1 2023-10-11T11:06:13.716 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:06:13.716 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:06:13.716 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_27 2023-10-11T11:06:13.716 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' snapshot = journal ']' 2023-10-11T11:06:13.717 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' snapshot = snapshot ']' 2023-10-11T11:06:13.717 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_27 2023-10-11T11:06:13.717 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local local_cluster=cluster1 2023-10-11T11:06:13.717 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:06:13.718 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:06:13.718 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_27 2023-10-11T11:06:13.718 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ mkfname cluster2-mirror-image_27.status 2023-10-11T11:06:13.718 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ echo cluster2-mirror-image_27.status 2023-10-11T11:06:13.719 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -e 's|[/ ]|_|g' 2023-10-11T11:06:13.719 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local status_log=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_27.status 2023-10-11T11:06:13.719 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ mkfname cluster1-mirror-image_27.status 2023-10-11T11:06:13.719 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ echo cluster1-mirror-image_27.status 2023-10-11T11:06:13.720 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -e 's|[/ ]|_|g' 2023-10-11T11:06:13.721 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local local_status_log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_27.status 2023-10-11T11:06:13.721 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ mirror_image_snapshot cluster2 mirror image_27 2023-10-11T11:06:13.721 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:06:13.721 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:06:13.722 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_27 2023-10-11T11:06:13.722 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_27 2023-10-11T11:06:14.419 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:Snapshot ID: 171 2023-10-11T11:06:14.430 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_27 /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_27.status 2023-10-11T11:06:14.431 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:06:14.431 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:06:14.431 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_27 2023-10-11T11:06:14.431 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_27.status 2023-10-11T11:06:14.431 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_27 --format xml 2023-10-11T11:06:14.432 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:06:14.474 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-10-11T11:06:14.476 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snapshot_id=171 2023-10-11T11:06:14.476 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T11:06:14.477 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T11:06:14.477 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.2 2023-10-11T11:06:14.678 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_27 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_27.status 2023-10-11T11:06:14.678 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:06:14.678 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:06:14.679 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_27 2023-10-11T11:06:14.679 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_27.status 2023-10-11T11:06:14.679 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_27 --format xml 2023-10-11T11:06:14.679 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:06:14.728 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T11:06:14.728 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T11:06:14.730 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:-:1.1: Document is empty 2023-10-11T11:06:14.730 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: 2023-10-11T11:06:14.730 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:^ 2023-10-11T11:06:14.730 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id= 2023-10-11T11:06:14.731 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 171 = '' 2023-10-11T11:06:14.731 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T11:06:14.731 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.4 2023-10-11T11:06:15.131 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_27 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_27.status 2023-10-11T11:06:15.132 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:06:15.132 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:06:15.132 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_27 2023-10-11T11:06:15.132 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_27.status 2023-10-11T11:06:15.133 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_27 --format xml 2023-10-11T11:06:15.133 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:06:15.178 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T11:06:15.179 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T11:06:15.180 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:-:1.1: Document is empty 2023-10-11T11:06:15.180 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: 2023-10-11T11:06:15.181 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:^ 2023-10-11T11:06:15.181 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id= 2023-10-11T11:06:15.181 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 171 = '' 2023-10-11T11:06:15.181 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T11:06:15.182 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.8 2023-10-11T11:06:15.981 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_27 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_27.status 2023-10-11T11:06:15.981 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:06:15.981 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:06:15.982 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_27 2023-10-11T11:06:15.982 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_27.status 2023-10-11T11:06:15.982 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_27 --format xml 2023-10-11T11:06:15.982 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:06:16.026 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T11:06:16.028 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id=171 2023-10-11T11:06:16.029 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 171 = 171 2023-10-11T11:06:16.029 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:06:16.029 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_snap_present cluster1 mirror image_27 snap 2023-10-11T11:06:16.029 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:06:16.029 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:06:16.029 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_27 2023-10-11T11:06:16.030 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap 2023-10-11T11:06:16.030 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T11:06:16.030 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-10-11T11:06:16.030 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 1 2023-10-11T11:06:17.029 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 info mirror/image_27@snap 2023-10-11T11:06:17.076 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:rbd image 'image_27': 2023-10-11T11:06:17.076 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: size 128 MiB in 32 objects 2023-10-11T11:06:17.076 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: order 22 (4 MiB objects) 2023-10-11T11:06:17.076 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: snapshot_count: 2 2023-10-11T11:06:17.077 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: id: 1289743d36f4 2023-10-11T11:06:17.077 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: block_name_prefix: rbd_data.1289743d36f4 2023-10-11T11:06:17.077 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: format: 2 2023-10-11T11:06:17.077 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-10-11T11:06:17.077 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: op_features: 2023-10-11T11:06:17.077 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: flags: 2023-10-11T11:06:17.078 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: create_timestamp: Wed Oct 11 10:58:39 2023 2023-10-11T11:06:17.078 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: access_timestamp: Wed Oct 11 10:58:39 2023 2023-10-11T11:06:17.078 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: modify_timestamp: Wed Oct 11 10:58:39 2023 2023-10-11T11:06:17.078 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: protected: False 2023-10-11T11:06:17.078 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring state: enabled 2023-10-11T11:06:17.079 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring mode: snapshot 2023-10-11T11:06:17.079 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring global id: fed85990-e790-4702-a20f-70d066da94cd 2023-10-11T11:06:17.079 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring primary: false 2023-10-11T11:06:17.080 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:06:17.080 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ compare_image_snaps mirror image_27 snap 2023-10-11T11:06:17.080 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:06:17.080 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_27 2023-10-11T11:06:17.081 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap 2023-10-11T11:06:17.081 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local ret=0 2023-10-11T11:06:17.081 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local rmt_export=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_27.export 2023-10-11T11:06:17.081 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local loc_export=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_27.export 2023-10-11T11:06:17.081 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rm -f /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_27.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_27.export 2023-10-11T11:06:17.082 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 -p mirror export image_27@snap /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_27.export 2023-10-11T11:06:17.252 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-10-11T11:06:17.258 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 -p mirror export image_27@snap /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_27.export 2023-10-11T11:06:17.497 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-10-11T11:06:17.503 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ cmp /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_27.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_27.export 2023-10-11T11:06:17.558 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rm -f /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_27.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_27.export 2023-10-11T11:06:17.594 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:06:17.594 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T11:06:17.595 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_28 2023-10-11T11:06:17.595 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ create_snap cluster2 mirror image_28 snap 2023-10-11T11:06:17.595 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:06:17.595 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:06:17.595 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_28 2023-10-11T11:06:17.596 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap 2023-10-11T11:06:17.596 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 -p mirror snap create image_28@snap --debug-rbd=20 --debug-journaler=20 2023-10-11T11:06:19.518 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_replay_started cluster1 mirror image_28 2023-10-11T11:06:19.518 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:06:19.518 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:06:19.519 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_28 2023-10-11T11:06:19.519 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_replay_state cluster1 mirror image_28 started 2023-10-11T11:06:19.519 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:06:19.519 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:06:19.519 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_28 2023-10-11T11:06:19.520 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local state=started 2023-10-11T11:06:19.520 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T11:06:19.520 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-10-11T11:06:19.520 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 1 2023-10-11T11:06:20.519 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test_image_replay_state cluster1 mirror image_28 started 2023-10-11T11:06:20.519 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:06:20.520 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:06:20.520 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_28 2023-10-11T11:06:20.520 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local test_state=started 2023-10-11T11:06:20.520 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local status_result 2023-10-11T11:06:20.520 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_state=stopped 2023-10-11T11:06:20.521 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_28 2023-10-11T11:06:20.521 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_instance=cluster1 2023-10-11T11:06:20.521 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T11:06:20.521 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ grep -i state 2023-10-11T11:06:20.521 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:06:20.521 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance=cluster1 2023-10-11T11:06:20.522 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local loop_instance 2023-10-11T11:06:20.522 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-10-11T11:06:20.522 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sleep 0 2023-10-11T11:06:20.525 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-10-11T11:06:20.525 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ seq 0 3 2023-10-11T11:06:20.525 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-10-11T11:06:20.525 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_28 2023-10-11T11:06:20.525 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1:0 2023-10-11T11:06:20.525 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T11:06:20.526 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance 2023-10-11T11:06:20.526 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-10-11T11:06:20.526 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local val=cluster1:0 2023-10-11T11:06:20.526 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_var_name=cluster 2023-10-11T11:06:20.526 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance_var_name=instance 2023-10-11T11:06:20.526 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ cluster=cluster1 2023-10-11T11:06:20.527 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ instance=0 2023-10-11T11:06:20.527 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' 0 = cluster1:0 ']' 2023-10-11T11:06:20.527 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval cluster=cluster1 2023-10-11T11:06:20.527 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:06:20.527 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval instance=0 2023-10-11T11:06:20.528 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=0 2023-10-11T11:06:20.528 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-10-11T11:06:20.528 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local local_cluster=cluster1:0 2023-10-11T11:06:20.528 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster=cluster1 2023-10-11T11:06:20.528 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance 2023-10-11T11:06:20.528 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-10-11T11:06:20.529 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local val=cluster1:0 2023-10-11T11:06:20.529 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster_var_name=local_cluster 2023-10-11T11:06:20.529 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance_var_name=instance 2023-10-11T11:06:20.529 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:06:20.529 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=0 2023-10-11T11:06:20.529 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ '[' 0 = cluster1:0 ']' 2023-10-11T11:06:20.530 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval local_cluster=cluster1 2023-10-11T11:06:20.530 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ local_cluster=cluster1 2023-10-11T11:06:20.530 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval instance=0 2023-10-11T11:06:20.530 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ instance=0 2023-10-11T11:06:20.530 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-10-11T11:06:20.546 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T11:06:20.546 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T11:06:20.546 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T11:06:20.546 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_28 2023-10-11T11:06:20.646 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:Invalid command: unused arguments: ['mirror/image_28'] 2023-10-11T11:06:20.647 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd mirror status : get status for rbd mirror 2023-10-11T11:06:20.647 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:admin_socket: invalid command 2023-10-11T11:06:20.650 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-10-11T11:06:20.650 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_28 2023-10-11T11:06:20.650 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1:1 2023-10-11T11:06:20.650 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T11:06:20.651 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance 2023-10-11T11:06:20.651 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-10-11T11:06:20.651 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local val=cluster1:1 2023-10-11T11:06:20.651 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_var_name=cluster 2023-10-11T11:06:20.651 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance_var_name=instance 2023-10-11T11:06:20.651 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ cluster=cluster1 2023-10-11T11:06:20.652 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ instance=1 2023-10-11T11:06:20.652 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' 1 = cluster1:1 ']' 2023-10-11T11:06:20.652 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval cluster=cluster1 2023-10-11T11:06:20.652 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:06:20.652 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval instance=1 2023-10-11T11:06:20.653 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=1 2023-10-11T11:06:20.653 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-10-11T11:06:20.653 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local local_cluster=cluster1:1 2023-10-11T11:06:20.653 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster=cluster1 2023-10-11T11:06:20.653 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance 2023-10-11T11:06:20.653 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-10-11T11:06:20.654 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local val=cluster1:1 2023-10-11T11:06:20.654 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster_var_name=local_cluster 2023-10-11T11:06:20.654 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance_var_name=instance 2023-10-11T11:06:20.654 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:06:20.654 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=1 2023-10-11T11:06:20.655 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ '[' 1 = cluster1:1 ']' 2023-10-11T11:06:20.655 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval local_cluster=cluster1 2023-10-11T11:06:20.655 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ local_cluster=cluster1 2023-10-11T11:06:20.655 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval instance=1 2023-10-11T11:06:20.655 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ instance=1 2023-10-11T11:06:20.655 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-10-11T11:06:20.669 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T11:06:20.669 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T11:06:20.669 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T11:06:20.669 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_28 2023-10-11T11:06:20.819 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ return 0 2023-10-11T11:06:20.820 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ status_result=' "state": "Replaying"' 2023-10-11T11:06:20.821 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ echo ' "state": "Replaying"' 2023-10-11T11:06:20.821 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ grep -i Replaying 2023-10-11T11:06:20.821 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: "state": "Replaying" 2023-10-11T11:06:20.824 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ current_state=started 2023-10-11T11:06:20.824 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test started = started 2023-10-11T11:06:20.824 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:06:20.825 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_28 2023-10-11T11:06:20.825 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local local_cluster=cluster1 2023-10-11T11:06:20.825 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:06:20.825 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:06:20.825 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_28 2023-10-11T11:06:20.826 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' snapshot = journal ']' 2023-10-11T11:06:20.826 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' snapshot = snapshot ']' 2023-10-11T11:06:20.826 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_28 2023-10-11T11:06:20.826 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local local_cluster=cluster1 2023-10-11T11:06:20.827 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:06:20.827 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:06:20.827 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_28 2023-10-11T11:06:20.827 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ mkfname cluster2-mirror-image_28.status 2023-10-11T11:06:20.827 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ echo cluster2-mirror-image_28.status 2023-10-11T11:06:20.828 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -e 's|[/ ]|_|g' 2023-10-11T11:06:20.828 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local status_log=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_28.status 2023-10-11T11:06:20.828 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ mkfname cluster1-mirror-image_28.status 2023-10-11T11:06:20.828 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ echo cluster1-mirror-image_28.status 2023-10-11T11:06:20.828 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -e 's|[/ ]|_|g' 2023-10-11T11:06:20.830 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local local_status_log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_28.status 2023-10-11T11:06:20.830 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ mirror_image_snapshot cluster2 mirror image_28 2023-10-11T11:06:20.830 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:06:20.830 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:06:20.831 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_28 2023-10-11T11:06:20.831 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_28 2023-10-11T11:06:21.520 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:Snapshot ID: 173 2023-10-11T11:06:21.532 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_28 /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_28.status 2023-10-11T11:06:21.532 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:06:21.532 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:06:21.533 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_28 2023-10-11T11:06:21.533 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_28.status 2023-10-11T11:06:21.533 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_28 --format xml 2023-10-11T11:06:21.533 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:06:21.576 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-10-11T11:06:21.578 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snapshot_id=173 2023-10-11T11:06:21.578 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T11:06:21.578 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T11:06:21.578 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.2 2023-10-11T11:06:21.779 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_28 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_28.status 2023-10-11T11:06:21.780 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:06:21.780 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:06:21.780 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_28 2023-10-11T11:06:21.780 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_28.status 2023-10-11T11:06:21.780 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_28 --format xml 2023-10-11T11:06:21.781 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:06:21.826 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T11:06:21.826 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T11:06:21.827 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:-:1.1: Document is empty 2023-10-11T11:06:21.827 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: 2023-10-11T11:06:21.828 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:^ 2023-10-11T11:06:21.828 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id= 2023-10-11T11:06:21.829 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 173 = '' 2023-10-11T11:06:21.829 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T11:06:21.829 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.4 2023-10-11T11:06:22.229 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_28 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_28.status 2023-10-11T11:06:22.229 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:06:22.229 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:06:22.230 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_28 2023-10-11T11:06:22.230 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_28.status 2023-10-11T11:06:22.230 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_28 --format xml 2023-10-11T11:06:22.230 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:06:22.276 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T11:06:22.276 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T11:06:22.278 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:-:1.1: Document is empty 2023-10-11T11:06:22.278 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: 2023-10-11T11:06:22.278 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:^ 2023-10-11T11:06:22.279 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id= 2023-10-11T11:06:22.279 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 173 = '' 2023-10-11T11:06:22.279 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T11:06:22.279 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.8 2023-10-11T11:06:23.080 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_28 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_28.status 2023-10-11T11:06:23.080 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:06:23.080 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:06:23.080 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_28 2023-10-11T11:06:23.080 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_28.status 2023-10-11T11:06:23.081 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_28 --format xml 2023-10-11T11:06:23.081 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:06:23.123 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T11:06:23.126 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id=173 2023-10-11T11:06:23.126 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 173 = 173 2023-10-11T11:06:23.126 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:06:23.126 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_snap_present cluster1 mirror image_28 snap 2023-10-11T11:06:23.126 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:06:23.127 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:06:23.127 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_28 2023-10-11T11:06:23.127 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap 2023-10-11T11:06:23.127 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T11:06:23.127 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-10-11T11:06:23.127 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 1 2023-10-11T11:06:24.126 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 info mirror/image_28@snap 2023-10-11T11:06:24.175 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:rbd image 'image_28': 2023-10-11T11:06:24.175 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: size 128 MiB in 32 objects 2023-10-11T11:06:24.175 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: order 22 (4 MiB objects) 2023-10-11T11:06:24.176 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: snapshot_count: 2 2023-10-11T11:06:24.176 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: id: 12893d1c381f 2023-10-11T11:06:24.176 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: block_name_prefix: rbd_data.12893d1c381f 2023-10-11T11:06:24.176 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: format: 2 2023-10-11T11:06:24.176 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-10-11T11:06:24.177 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: op_features: 2023-10-11T11:06:24.177 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: flags: 2023-10-11T11:06:24.177 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: create_timestamp: Wed Oct 11 10:58:48 2023 2023-10-11T11:06:24.177 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: access_timestamp: Wed Oct 11 10:58:48 2023 2023-10-11T11:06:24.177 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: modify_timestamp: Wed Oct 11 10:58:48 2023 2023-10-11T11:06:24.178 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: protected: False 2023-10-11T11:06:24.178 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring state: enabled 2023-10-11T11:06:24.178 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring mode: snapshot 2023-10-11T11:06:24.178 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring global id: 48b201ee-f8ea-4845-97ba-749a1f35906b 2023-10-11T11:06:24.178 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring primary: false 2023-10-11T11:06:24.179 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:06:24.180 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ compare_image_snaps mirror image_28 snap 2023-10-11T11:06:24.180 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:06:24.180 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_28 2023-10-11T11:06:24.180 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap 2023-10-11T11:06:24.180 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local ret=0 2023-10-11T11:06:24.181 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local rmt_export=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_28.export 2023-10-11T11:06:24.181 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local loc_export=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_28.export 2023-10-11T11:06:24.181 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rm -f /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_28.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_28.export 2023-10-11T11:06:24.181 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 -p mirror export image_28@snap /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_28.export 2023-10-11T11:06:24.365 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-10-11T11:06:24.370 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 -p mirror export image_28@snap /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_28.export 2023-10-11T11:06:24.628 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-10-11T11:06:24.634 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ cmp /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_28.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_28.export 2023-10-11T11:06:24.690 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rm -f /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_28.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_28.export 2023-10-11T11:06:24.727 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:06:24.727 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T11:06:24.727 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_29 2023-10-11T11:06:24.727 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ create_snap cluster2 mirror image_29 snap 2023-10-11T11:06:24.727 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:06:24.727 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:06:24.728 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_29 2023-10-11T11:06:24.728 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap 2023-10-11T11:06:24.728 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 -p mirror snap create image_29@snap --debug-rbd=20 --debug-journaler=20 2023-10-11T11:06:26.531 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_replay_started cluster1 mirror image_29 2023-10-11T11:06:26.531 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:06:26.531 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:06:26.531 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_29 2023-10-11T11:06:26.532 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_replay_state cluster1 mirror image_29 started 2023-10-11T11:06:26.532 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:06:26.532 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:06:26.532 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_29 2023-10-11T11:06:26.532 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local state=started 2023-10-11T11:06:26.533 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T11:06:26.533 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-10-11T11:06:26.533 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 1 2023-10-11T11:06:27.532 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test_image_replay_state cluster1 mirror image_29 started 2023-10-11T11:06:27.532 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:06:27.533 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:06:27.533 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_29 2023-10-11T11:06:27.533 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local test_state=started 2023-10-11T11:06:27.533 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local status_result 2023-10-11T11:06:27.533 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_state=stopped 2023-10-11T11:06:27.535 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_29 2023-10-11T11:06:27.535 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_instance=cluster1 2023-10-11T11:06:27.535 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ grep -i state 2023-10-11T11:06:27.535 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T11:06:27.535 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:06:27.536 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance=cluster1 2023-10-11T11:06:27.536 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local loop_instance 2023-10-11T11:06:27.536 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-10-11T11:06:27.536 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sleep 0 2023-10-11T11:06:27.536 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-10-11T11:06:27.537 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ seq 0 3 2023-10-11T11:06:27.539 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-10-11T11:06:27.539 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_29 2023-10-11T11:06:27.540 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1:0 2023-10-11T11:06:27.540 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T11:06:27.540 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance 2023-10-11T11:06:27.540 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-10-11T11:06:27.540 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local val=cluster1:0 2023-10-11T11:06:27.541 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_var_name=cluster 2023-10-11T11:06:27.541 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance_var_name=instance 2023-10-11T11:06:27.541 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ cluster=cluster1 2023-10-11T11:06:27.541 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ instance=0 2023-10-11T11:06:27.541 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' 0 = cluster1:0 ']' 2023-10-11T11:06:27.541 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval cluster=cluster1 2023-10-11T11:06:27.542 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:06:27.542 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval instance=0 2023-10-11T11:06:27.542 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=0 2023-10-11T11:06:27.542 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-10-11T11:06:27.542 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local local_cluster=cluster1:0 2023-10-11T11:06:27.543 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster=cluster1 2023-10-11T11:06:27.543 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance 2023-10-11T11:06:27.543 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-10-11T11:06:27.543 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local val=cluster1:0 2023-10-11T11:06:27.543 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster_var_name=local_cluster 2023-10-11T11:06:27.543 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance_var_name=instance 2023-10-11T11:06:27.544 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:06:27.544 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=0 2023-10-11T11:06:27.544 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ '[' 0 = cluster1:0 ']' 2023-10-11T11:06:27.544 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval local_cluster=cluster1 2023-10-11T11:06:27.544 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ local_cluster=cluster1 2023-10-11T11:06:27.544 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval instance=0 2023-10-11T11:06:27.545 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ instance=0 2023-10-11T11:06:27.545 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-10-11T11:06:27.559 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T11:06:27.559 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T11:06:27.559 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T11:06:27.559 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_29 2023-10-11T11:06:27.660 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:Invalid command: unused arguments: ['mirror/image_29'] 2023-10-11T11:06:27.660 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd mirror status : get status for rbd mirror 2023-10-11T11:06:27.660 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:admin_socket: invalid command 2023-10-11T11:06:27.663 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-10-11T11:06:27.663 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_29 2023-10-11T11:06:27.663 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1:1 2023-10-11T11:06:27.663 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T11:06:27.663 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance 2023-10-11T11:06:27.663 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-10-11T11:06:27.664 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local val=cluster1:1 2023-10-11T11:06:27.664 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_var_name=cluster 2023-10-11T11:06:27.664 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance_var_name=instance 2023-10-11T11:06:27.664 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ cluster=cluster1 2023-10-11T11:06:27.664 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ instance=1 2023-10-11T11:06:27.664 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' 1 = cluster1:1 ']' 2023-10-11T11:06:27.665 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval cluster=cluster1 2023-10-11T11:06:27.665 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:06:27.665 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval instance=1 2023-10-11T11:06:27.665 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=1 2023-10-11T11:06:27.665 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-10-11T11:06:27.666 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local local_cluster=cluster1:1 2023-10-11T11:06:27.666 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster=cluster1 2023-10-11T11:06:27.666 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance 2023-10-11T11:06:27.666 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-10-11T11:06:27.666 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local val=cluster1:1 2023-10-11T11:06:27.667 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster_var_name=local_cluster 2023-10-11T11:06:27.667 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance_var_name=instance 2023-10-11T11:06:27.667 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:06:27.667 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=1 2023-10-11T11:06:27.667 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ '[' 1 = cluster1:1 ']' 2023-10-11T11:06:27.667 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval local_cluster=cluster1 2023-10-11T11:06:27.668 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ local_cluster=cluster1 2023-10-11T11:06:27.668 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval instance=1 2023-10-11T11:06:27.668 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ instance=1 2023-10-11T11:06:27.668 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-10-11T11:06:27.682 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T11:06:27.682 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T11:06:27.682 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T11:06:27.682 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_29 2023-10-11T11:06:27.829 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ return 0 2023-10-11T11:06:27.830 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ status_result=' "state": "Replaying"' 2023-10-11T11:06:27.830 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ echo ' "state": "Replaying"' 2023-10-11T11:06:27.830 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ grep -i Replaying 2023-10-11T11:06:27.831 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: "state": "Replaying" 2023-10-11T11:06:27.833 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ current_state=started 2023-10-11T11:06:27.833 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test started = started 2023-10-11T11:06:27.833 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:06:27.833 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_29 2023-10-11T11:06:27.833 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local local_cluster=cluster1 2023-10-11T11:06:27.834 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:06:27.834 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:06:27.834 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_29 2023-10-11T11:06:27.834 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' snapshot = journal ']' 2023-10-11T11:06:27.834 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' snapshot = snapshot ']' 2023-10-11T11:06:27.835 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_29 2023-10-11T11:06:27.835 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local local_cluster=cluster1 2023-10-11T11:06:27.835 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:06:27.835 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:06:27.835 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_29 2023-10-11T11:06:27.835 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ mkfname cluster2-mirror-image_29.status 2023-10-11T11:06:27.836 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ echo cluster2-mirror-image_29.status 2023-10-11T11:06:27.836 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -e 's|[/ ]|_|g' 2023-10-11T11:06:27.837 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local status_log=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_29.status 2023-10-11T11:06:27.837 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ mkfname cluster1-mirror-image_29.status 2023-10-11T11:06:27.837 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ echo cluster1-mirror-image_29.status 2023-10-11T11:06:27.838 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -e 's|[/ ]|_|g' 2023-10-11T11:06:27.838 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local local_status_log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_29.status 2023-10-11T11:06:27.838 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ mirror_image_snapshot cluster2 mirror image_29 2023-10-11T11:06:27.838 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:06:27.838 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:06:27.839 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_29 2023-10-11T11:06:27.839 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_29 2023-10-11T11:06:28.533 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:Snapshot ID: 175 2023-10-11T11:06:28.545 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_29 /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_29.status 2023-10-11T11:06:28.546 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:06:28.546 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:06:28.546 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_29 2023-10-11T11:06:28.546 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_29.status 2023-10-11T11:06:28.547 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_29 --format xml 2023-10-11T11:06:28.547 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:06:28.589 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-10-11T11:06:28.592 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snapshot_id=175 2023-10-11T11:06:28.592 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T11:06:28.592 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T11:06:28.592 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.2 2023-10-11T11:06:28.793 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_29 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_29.status 2023-10-11T11:06:28.793 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:06:28.793 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:06:28.793 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_29 2023-10-11T11:06:28.794 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_29.status 2023-10-11T11:06:28.794 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_29 --format xml 2023-10-11T11:06:28.794 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:06:28.839 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T11:06:28.839 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T11:06:28.840 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:-:1.1: Document is empty 2023-10-11T11:06:28.840 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: 2023-10-11T11:06:28.841 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:^ 2023-10-11T11:06:28.841 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id= 2023-10-11T11:06:28.841 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 175 = '' 2023-10-11T11:06:28.842 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T11:06:28.842 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.4 2023-10-11T11:06:29.242 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_29 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_29.status 2023-10-11T11:06:29.242 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:06:29.243 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:06:29.243 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_29 2023-10-11T11:06:29.243 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_29.status 2023-10-11T11:06:29.243 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_29 --format xml 2023-10-11T11:06:29.243 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:06:29.291 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T11:06:29.291 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T11:06:29.292 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:-:1.1: Document is empty 2023-10-11T11:06:29.293 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: 2023-10-11T11:06:29.293 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:^ 2023-10-11T11:06:29.293 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id= 2023-10-11T11:06:29.294 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 175 = '' 2023-10-11T11:06:29.294 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T11:06:29.294 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.8 2023-10-11T11:06:30.094 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_29 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_29.status 2023-10-11T11:06:30.095 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:06:30.095 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:06:30.095 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_29 2023-10-11T11:06:30.095 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_29.status 2023-10-11T11:06:30.096 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_29 --format xml 2023-10-11T11:06:30.096 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:06:30.139 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T11:06:30.141 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id=175 2023-10-11T11:06:30.141 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 175 = 175 2023-10-11T11:06:30.142 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:06:30.142 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_snap_present cluster1 mirror image_29 snap 2023-10-11T11:06:30.142 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:06:30.142 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:06:30.142 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_29 2023-10-11T11:06:30.142 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap 2023-10-11T11:06:30.143 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T11:06:30.143 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-10-11T11:06:30.143 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 1 2023-10-11T11:06:31.142 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 info mirror/image_29@snap 2023-10-11T11:06:31.185 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:rbd image 'image_29': 2023-10-11T11:06:31.185 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: size 128 MiB in 32 objects 2023-10-11T11:06:31.186 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: order 22 (4 MiB objects) 2023-10-11T11:06:31.186 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: snapshot_count: 2 2023-10-11T11:06:31.186 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: id: 12899d91ec5f 2023-10-11T11:06:31.186 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: block_name_prefix: rbd_data.12899d91ec5f 2023-10-11T11:06:31.187 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: format: 2 2023-10-11T11:06:31.187 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-10-11T11:06:31.187 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: op_features: 2023-10-11T11:06:31.187 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: flags: 2023-10-11T11:06:31.187 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: create_timestamp: Wed Oct 11 10:58:54 2023 2023-10-11T11:06:31.188 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: access_timestamp: Wed Oct 11 10:58:54 2023 2023-10-11T11:06:31.188 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: modify_timestamp: Wed Oct 11 10:58:54 2023 2023-10-11T11:06:31.188 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: protected: False 2023-10-11T11:06:31.188 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring state: enabled 2023-10-11T11:06:31.188 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring mode: snapshot 2023-10-11T11:06:31.189 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring global id: d589d227-75bc-45bf-a707-46d8d0068324 2023-10-11T11:06:31.189 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring primary: false 2023-10-11T11:06:31.190 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:06:31.190 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ compare_image_snaps mirror image_29 snap 2023-10-11T11:06:31.190 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:06:31.191 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_29 2023-10-11T11:06:31.191 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap 2023-10-11T11:06:31.191 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local ret=0 2023-10-11T11:06:31.191 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local rmt_export=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_29.export 2023-10-11T11:06:31.191 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local loc_export=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_29.export 2023-10-11T11:06:31.192 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rm -f /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_29.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_29.export 2023-10-11T11:06:31.192 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 -p mirror export image_29@snap /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_29.export 2023-10-11T11:06:31.364 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-10-11T11:06:31.370 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 -p mirror export image_29@snap /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_29.export 2023-10-11T11:06:31.607 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-10-11T11:06:31.612 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ cmp /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_29.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_29.export 2023-10-11T11:06:31.667 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rm -f /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_29.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_29.export 2023-10-11T11:06:31.703 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:06:31.703 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T11:06:31.704 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_30 2023-10-11T11:06:31.704 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ create_snap cluster2 mirror image_30 snap 2023-10-11T11:06:31.704 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:06:31.704 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:06:31.704 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_30 2023-10-11T11:06:31.705 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap 2023-10-11T11:06:31.705 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 -p mirror snap create image_30@snap --debug-rbd=20 --debug-journaler=20 2023-10-11T11:06:33.547 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_replay_started cluster1 mirror image_30 2023-10-11T11:06:33.547 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:06:33.547 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:06:33.548 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_30 2023-10-11T11:06:33.548 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_replay_state cluster1 mirror image_30 started 2023-10-11T11:06:33.548 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:06:33.549 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:06:33.549 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_30 2023-10-11T11:06:33.549 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local state=started 2023-10-11T11:06:33.549 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T11:06:33.550 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-10-11T11:06:33.550 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 1 2023-10-11T11:06:34.547 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test_image_replay_state cluster1 mirror image_30 started 2023-10-11T11:06:34.548 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:06:34.548 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:06:34.548 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_30 2023-10-11T11:06:34.548 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local test_state=started 2023-10-11T11:06:34.549 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local status_result 2023-10-11T11:06:34.549 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_state=stopped 2023-10-11T11:06:34.550 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_30 2023-10-11T11:06:34.550 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ grep -i state 2023-10-11T11:06:34.550 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_instance=cluster1 2023-10-11T11:06:34.550 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T11:06:34.551 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:06:34.551 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance=cluster1 2023-10-11T11:06:34.551 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local loop_instance 2023-10-11T11:06:34.551 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-10-11T11:06:34.551 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sleep 0 2023-10-11T11:06:34.551 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-10-11T11:06:34.552 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ seq 0 3 2023-10-11T11:06:34.553 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-10-11T11:06:34.553 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_30 2023-10-11T11:06:34.553 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1:0 2023-10-11T11:06:34.554 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T11:06:34.554 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance 2023-10-11T11:06:34.554 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-10-11T11:06:34.554 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local val=cluster1:0 2023-10-11T11:06:34.554 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_var_name=cluster 2023-10-11T11:06:34.555 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance_var_name=instance 2023-10-11T11:06:34.555 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ cluster=cluster1 2023-10-11T11:06:34.555 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ instance=0 2023-10-11T11:06:34.555 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' 0 = cluster1:0 ']' 2023-10-11T11:06:34.555 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval cluster=cluster1 2023-10-11T11:06:34.555 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:06:34.556 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval instance=0 2023-10-11T11:06:34.556 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=0 2023-10-11T11:06:34.556 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-10-11T11:06:34.556 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local local_cluster=cluster1:0 2023-10-11T11:06:34.556 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster=cluster1 2023-10-11T11:06:34.556 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance 2023-10-11T11:06:34.558 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-10-11T11:06:34.558 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local val=cluster1:0 2023-10-11T11:06:34.558 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster_var_name=local_cluster 2023-10-11T11:06:34.558 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance_var_name=instance 2023-10-11T11:06:34.559 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:06:34.559 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=0 2023-10-11T11:06:34.559 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ '[' 0 = cluster1:0 ']' 2023-10-11T11:06:34.559 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval local_cluster=cluster1 2023-10-11T11:06:34.559 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ local_cluster=cluster1 2023-10-11T11:06:34.559 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval instance=0 2023-10-11T11:06:34.560 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ instance=0 2023-10-11T11:06:34.560 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-10-11T11:06:34.575 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T11:06:34.575 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T11:06:34.575 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T11:06:34.576 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_30 2023-10-11T11:06:34.677 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:Invalid command: unused arguments: ['mirror/image_30'] 2023-10-11T11:06:34.677 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd mirror status : get status for rbd mirror 2023-10-11T11:06:34.678 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:admin_socket: invalid command 2023-10-11T11:06:34.680 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-10-11T11:06:34.680 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_30 2023-10-11T11:06:34.681 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1:1 2023-10-11T11:06:34.681 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T11:06:34.681 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance 2023-10-11T11:06:34.681 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-10-11T11:06:34.681 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local val=cluster1:1 2023-10-11T11:06:34.682 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_var_name=cluster 2023-10-11T11:06:34.682 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance_var_name=instance 2023-10-11T11:06:34.682 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ cluster=cluster1 2023-10-11T11:06:34.682 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ instance=1 2023-10-11T11:06:34.682 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' 1 = cluster1:1 ']' 2023-10-11T11:06:34.682 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval cluster=cluster1 2023-10-11T11:06:34.683 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:06:34.683 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval instance=1 2023-10-11T11:06:34.683 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=1 2023-10-11T11:06:34.683 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-10-11T11:06:34.683 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local local_cluster=cluster1:1 2023-10-11T11:06:34.683 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster=cluster1 2023-10-11T11:06:34.684 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance 2023-10-11T11:06:34.684 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-10-11T11:06:34.684 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local val=cluster1:1 2023-10-11T11:06:34.684 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster_var_name=local_cluster 2023-10-11T11:06:34.684 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance_var_name=instance 2023-10-11T11:06:34.685 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:06:34.685 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=1 2023-10-11T11:06:34.685 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ '[' 1 = cluster1:1 ']' 2023-10-11T11:06:34.685 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval local_cluster=cluster1 2023-10-11T11:06:34.685 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ local_cluster=cluster1 2023-10-11T11:06:34.685 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval instance=1 2023-10-11T11:06:34.686 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ instance=1 2023-10-11T11:06:34.686 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-10-11T11:06:34.700 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T11:06:34.700 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T11:06:34.700 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T11:06:34.700 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_30 2023-10-11T11:06:34.853 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ return 0 2023-10-11T11:06:34.853 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ status_result=' "state": "Replaying"' 2023-10-11T11:06:34.854 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ echo ' "state": "Replaying"' 2023-10-11T11:06:34.854 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ grep -i Replaying 2023-10-11T11:06:34.854 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: "state": "Replaying" 2023-10-11T11:06:34.856 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ current_state=started 2023-10-11T11:06:34.856 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test started = started 2023-10-11T11:06:34.857 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:06:34.857 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_30 2023-10-11T11:06:34.857 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local local_cluster=cluster1 2023-10-11T11:06:34.857 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:06:34.858 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:06:34.858 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_30 2023-10-11T11:06:34.858 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' snapshot = journal ']' 2023-10-11T11:06:34.858 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' snapshot = snapshot ']' 2023-10-11T11:06:34.858 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_30 2023-10-11T11:06:34.859 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local local_cluster=cluster1 2023-10-11T11:06:34.859 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:06:34.859 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:06:34.859 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_30 2023-10-11T11:06:34.859 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ mkfname cluster2-mirror-image_30.status 2023-10-11T11:06:34.859 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ echo cluster2-mirror-image_30.status 2023-10-11T11:06:34.860 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -e 's|[/ ]|_|g' 2023-10-11T11:06:34.860 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local status_log=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_30.status 2023-10-11T11:06:34.860 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ mkfname cluster1-mirror-image_30.status 2023-10-11T11:06:34.861 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ echo cluster1-mirror-image_30.status 2023-10-11T11:06:34.861 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -e 's|[/ ]|_|g' 2023-10-11T11:06:34.862 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local local_status_log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_30.status 2023-10-11T11:06:34.862 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ mirror_image_snapshot cluster2 mirror image_30 2023-10-11T11:06:34.862 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:06:34.862 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:06:34.862 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_30 2023-10-11T11:06:34.863 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_30 2023-10-11T11:06:35.550 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:Snapshot ID: 177 2023-10-11T11:06:35.561 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_30 /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_30.status 2023-10-11T11:06:35.562 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:06:35.562 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:06:35.562 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_30 2023-10-11T11:06:35.562 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_30.status 2023-10-11T11:06:35.562 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_30 --format xml 2023-10-11T11:06:35.563 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:06:35.606 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-10-11T11:06:35.608 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snapshot_id=177 2023-10-11T11:06:35.609 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T11:06:35.609 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T11:06:35.609 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.2 2023-10-11T11:06:35.810 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_30 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_30.status 2023-10-11T11:06:35.810 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:06:35.810 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:06:35.810 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_30 2023-10-11T11:06:35.810 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_30.status 2023-10-11T11:06:35.811 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_30 --format xml 2023-10-11T11:06:35.811 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:06:35.856 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T11:06:35.858 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id=97 2023-10-11T11:06:35.858 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 177 = 97 2023-10-11T11:06:35.858 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T11:06:35.858 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.4 2023-10-11T11:06:36.259 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_30 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_30.status 2023-10-11T11:06:36.259 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:06:36.260 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:06:36.260 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_30 2023-10-11T11:06:36.260 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_30.status 2023-10-11T11:06:36.260 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_30 --format xml 2023-10-11T11:06:36.260 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:06:36.307 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T11:06:36.308 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T11:06:36.309 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:-:1.1: Document is empty 2023-10-11T11:06:36.309 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: 2023-10-11T11:06:36.309 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:^ 2023-10-11T11:06:36.310 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id= 2023-10-11T11:06:36.310 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 177 = '' 2023-10-11T11:06:36.310 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T11:06:36.310 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.8 2023-10-11T11:06:37.111 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_30 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_30.status 2023-10-11T11:06:37.111 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:06:37.111 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:06:37.111 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_30 2023-10-11T11:06:37.112 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_30.status 2023-10-11T11:06:37.112 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_30 --format xml 2023-10-11T11:06:37.112 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:06:37.170 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T11:06:37.171 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T11:06:37.172 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:-:1.1: Document is empty 2023-10-11T11:06:37.172 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: 2023-10-11T11:06:37.172 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:^ 2023-10-11T11:06:37.173 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id= 2023-10-11T11:06:37.173 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 177 = '' 2023-10-11T11:06:37.173 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T11:06:37.174 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 1.6 2023-10-11T11:06:38.774 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_30 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_30.status 2023-10-11T11:06:38.774 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:06:38.775 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:06:38.775 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_30 2023-10-11T11:06:38.775 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_30.status 2023-10-11T11:06:38.775 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_30 --format xml 2023-10-11T11:06:38.775 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:06:38.826 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T11:06:38.829 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id=177 2023-10-11T11:06:38.829 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 177 = 177 2023-10-11T11:06:38.829 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:06:38.829 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_snap_present cluster1 mirror image_30 snap 2023-10-11T11:06:38.830 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:06:38.830 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:06:38.830 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_30 2023-10-11T11:06:38.830 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap 2023-10-11T11:06:38.830 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T11:06:38.831 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-10-11T11:06:38.831 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 1 2023-10-11T11:06:39.830 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 info mirror/image_30@snap 2023-10-11T11:06:39.879 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:rbd image 'image_30': 2023-10-11T11:06:39.879 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: size 128 MiB in 32 objects 2023-10-11T11:06:39.880 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: order 22 (4 MiB objects) 2023-10-11T11:06:39.880 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: snapshot_count: 2 2023-10-11T11:06:39.880 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: id: 12895e669f6b 2023-10-11T11:06:39.880 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: block_name_prefix: rbd_data.12895e669f6b 2023-10-11T11:06:39.880 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: format: 2 2023-10-11T11:06:39.880 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-10-11T11:06:39.881 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: op_features: 2023-10-11T11:06:39.881 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: flags: 2023-10-11T11:06:39.881 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: create_timestamp: Wed Oct 11 10:59:05 2023 2023-10-11T11:06:39.881 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: access_timestamp: Wed Oct 11 10:59:05 2023 2023-10-11T11:06:39.881 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: modify_timestamp: Wed Oct 11 10:59:05 2023 2023-10-11T11:06:39.882 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: protected: False 2023-10-11T11:06:39.882 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring state: enabled 2023-10-11T11:06:39.882 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring mode: snapshot 2023-10-11T11:06:39.882 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring global id: 3d66dfc7-26fa-4c9e-b7cc-ebb9eb40ba6a 2023-10-11T11:06:39.882 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring primary: false 2023-10-11T11:06:39.883 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:06:39.884 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ compare_image_snaps mirror image_30 snap 2023-10-11T11:06:39.884 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:06:39.884 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_30 2023-10-11T11:06:39.884 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap 2023-10-11T11:06:39.885 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local ret=0 2023-10-11T11:06:39.885 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local rmt_export=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_30.export 2023-10-11T11:06:39.885 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local loc_export=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_30.export 2023-10-11T11:06:39.885 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rm -f /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_30.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_30.export 2023-10-11T11:06:39.885 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 -p mirror export image_30@snap /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_30.export 2023-10-11T11:06:40.066 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-10-11T11:06:40.073 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 -p mirror export image_30@snap /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_30.export 2023-10-11T11:06:40.324 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-10-11T11:06:40.330 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ cmp /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_30.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_30.export 2023-10-11T11:06:40.386 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rm -f /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_30.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_30.export 2023-10-11T11:06:40.423 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:06:40.423 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T11:06:40.423 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_31 2023-10-11T11:06:40.423 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ create_snap cluster2 mirror image_31 snap 2023-10-11T11:06:40.423 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:06:40.424 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:06:40.424 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_31 2023-10-11T11:06:40.424 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap 2023-10-11T11:06:40.424 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 -p mirror snap create image_31@snap --debug-rbd=20 --debug-journaler=20 2023-10-11T11:06:42.418 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_replay_started cluster1 mirror image_31 2023-10-11T11:06:42.418 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:06:42.419 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:06:42.419 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_31 2023-10-11T11:06:42.419 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_replay_state cluster1 mirror image_31 started 2023-10-11T11:06:42.419 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:06:42.420 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:06:42.420 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_31 2023-10-11T11:06:42.420 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local state=started 2023-10-11T11:06:42.420 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T11:06:42.421 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-10-11T11:06:42.421 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 1 2023-10-11T11:06:43.419 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test_image_replay_state cluster1 mirror image_31 started 2023-10-11T11:06:43.419 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:06:43.419 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:06:43.419 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_31 2023-10-11T11:06:43.420 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local test_state=started 2023-10-11T11:06:43.420 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local status_result 2023-10-11T11:06:43.420 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_state=stopped 2023-10-11T11:06:43.421 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_31 2023-10-11T11:06:43.422 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_instance=cluster1 2023-10-11T11:06:43.422 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ grep -i state 2023-10-11T11:06:43.422 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T11:06:43.422 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:06:43.422 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance=cluster1 2023-10-11T11:06:43.423 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local loop_instance 2023-10-11T11:06:43.423 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-10-11T11:06:43.423 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sleep 0 2023-10-11T11:06:43.423 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-10-11T11:06:43.423 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ seq 0 3 2023-10-11T11:06:43.425 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-10-11T11:06:43.425 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_31 2023-10-11T11:06:43.425 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1:0 2023-10-11T11:06:43.425 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T11:06:43.426 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance 2023-10-11T11:06:43.426 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-10-11T11:06:43.426 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local val=cluster1:0 2023-10-11T11:06:43.426 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_var_name=cluster 2023-10-11T11:06:43.426 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance_var_name=instance 2023-10-11T11:06:43.427 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ cluster=cluster1 2023-10-11T11:06:43.427 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ instance=0 2023-10-11T11:06:43.427 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' 0 = cluster1:0 ']' 2023-10-11T11:06:43.427 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval cluster=cluster1 2023-10-11T11:06:43.427 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:06:43.428 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval instance=0 2023-10-11T11:06:43.428 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=0 2023-10-11T11:06:43.429 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-10-11T11:06:43.429 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local local_cluster=cluster1:0 2023-10-11T11:06:43.430 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster=cluster1 2023-10-11T11:06:43.430 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance 2023-10-11T11:06:43.430 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-10-11T11:06:43.430 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local val=cluster1:0 2023-10-11T11:06:43.430 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster_var_name=local_cluster 2023-10-11T11:06:43.430 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance_var_name=instance 2023-10-11T11:06:43.431 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:06:43.431 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=0 2023-10-11T11:06:43.431 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ '[' 0 = cluster1:0 ']' 2023-10-11T11:06:43.431 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval local_cluster=cluster1 2023-10-11T11:06:43.431 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ local_cluster=cluster1 2023-10-11T11:06:43.432 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval instance=0 2023-10-11T11:06:43.432 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ instance=0 2023-10-11T11:06:43.432 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-10-11T11:06:43.446 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T11:06:43.446 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T11:06:43.446 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T11:06:43.446 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_31 2023-10-11T11:06:43.547 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:Invalid command: unused arguments: ['mirror/image_31'] 2023-10-11T11:06:43.547 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd mirror status : get status for rbd mirror 2023-10-11T11:06:43.547 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:admin_socket: invalid command 2023-10-11T11:06:43.550 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-10-11T11:06:43.550 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_31 2023-10-11T11:06:43.550 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1:1 2023-10-11T11:06:43.551 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T11:06:43.551 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance 2023-10-11T11:06:43.551 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-10-11T11:06:43.551 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local val=cluster1:1 2023-10-11T11:06:43.551 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_var_name=cluster 2023-10-11T11:06:43.552 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance_var_name=instance 2023-10-11T11:06:43.552 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ cluster=cluster1 2023-10-11T11:06:43.552 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ instance=1 2023-10-11T11:06:43.552 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' 1 = cluster1:1 ']' 2023-10-11T11:06:43.552 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval cluster=cluster1 2023-10-11T11:06:43.552 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:06:43.553 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval instance=1 2023-10-11T11:06:43.553 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=1 2023-10-11T11:06:43.553 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-10-11T11:06:43.553 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local local_cluster=cluster1:1 2023-10-11T11:06:43.553 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster=cluster1 2023-10-11T11:06:43.553 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance 2023-10-11T11:06:43.554 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-10-11T11:06:43.554 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local val=cluster1:1 2023-10-11T11:06:43.554 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster_var_name=local_cluster 2023-10-11T11:06:43.554 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance_var_name=instance 2023-10-11T11:06:43.554 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:06:43.555 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=1 2023-10-11T11:06:43.555 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ '[' 1 = cluster1:1 ']' 2023-10-11T11:06:43.555 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval local_cluster=cluster1 2023-10-11T11:06:43.555 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ local_cluster=cluster1 2023-10-11T11:06:43.555 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval instance=1 2023-10-11T11:06:43.555 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ instance=1 2023-10-11T11:06:43.556 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-10-11T11:06:43.569 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T11:06:43.569 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T11:06:43.569 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T11:06:43.570 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_31 2023-10-11T11:06:43.717 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ return 0 2023-10-11T11:06:43.718 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ status_result=' "state": "Replaying"' 2023-10-11T11:06:43.718 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ echo ' "state": "Replaying"' 2023-10-11T11:06:43.718 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ grep -i Replaying 2023-10-11T11:06:43.719 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: "state": "Replaying" 2023-10-11T11:06:43.721 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ current_state=started 2023-10-11T11:06:43.721 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test started = started 2023-10-11T11:06:43.721 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:06:43.721 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_31 2023-10-11T11:06:43.721 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local local_cluster=cluster1 2023-10-11T11:06:43.722 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:06:43.722 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:06:43.722 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_31 2023-10-11T11:06:43.722 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' snapshot = journal ']' 2023-10-11T11:06:43.722 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' snapshot = snapshot ']' 2023-10-11T11:06:43.723 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_31 2023-10-11T11:06:43.723 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local local_cluster=cluster1 2023-10-11T11:06:43.723 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:06:43.723 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:06:43.723 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_31 2023-10-11T11:06:43.723 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ mkfname cluster2-mirror-image_31.status 2023-10-11T11:06:43.724 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ echo cluster2-mirror-image_31.status 2023-10-11T11:06:43.724 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -e 's|[/ ]|_|g' 2023-10-11T11:06:43.725 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local status_log=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_31.status 2023-10-11T11:06:43.725 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ mkfname cluster1-mirror-image_31.status 2023-10-11T11:06:43.725 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ echo cluster1-mirror-image_31.status 2023-10-11T11:06:43.726 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -e 's|[/ ]|_|g' 2023-10-11T11:06:43.726 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local local_status_log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_31.status 2023-10-11T11:06:43.726 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ mirror_image_snapshot cluster2 mirror image_31 2023-10-11T11:06:43.726 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:06:43.726 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:06:43.726 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_31 2023-10-11T11:06:43.727 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_31 2023-10-11T11:06:44.412 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:Snapshot ID: 179 2023-10-11T11:06:44.424 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_31 /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_31.status 2023-10-11T11:06:44.424 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:06:44.424 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:06:44.425 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_31 2023-10-11T11:06:44.425 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_31.status 2023-10-11T11:06:44.425 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_31 --format xml 2023-10-11T11:06:44.425 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:06:44.467 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-10-11T11:06:44.469 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snapshot_id=179 2023-10-11T11:06:44.469 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T11:06:44.469 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T11:06:44.470 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.2 2023-10-11T11:06:44.671 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_31 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_31.status 2023-10-11T11:06:44.671 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:06:44.671 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:06:44.671 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_31 2023-10-11T11:06:44.671 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_31.status 2023-10-11T11:06:44.672 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_31 --format xml 2023-10-11T11:06:44.672 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:06:44.718 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T11:06:44.718 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T11:06:44.719 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:-:1.1: Document is empty 2023-10-11T11:06:44.720 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: 2023-10-11T11:06:44.720 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:^ 2023-10-11T11:06:44.720 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id= 2023-10-11T11:06:44.721 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 179 = '' 2023-10-11T11:06:44.721 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T11:06:44.721 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.4 2023-10-11T11:06:45.121 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_31 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_31.status 2023-10-11T11:06:45.122 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:06:45.122 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:06:45.122 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_31 2023-10-11T11:06:45.122 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_31.status 2023-10-11T11:06:45.122 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_31 --format xml 2023-10-11T11:06:45.123 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:06:45.168 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T11:06:45.169 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T11:06:45.170 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:-:1.1: Document is empty 2023-10-11T11:06:45.170 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: 2023-10-11T11:06:45.171 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:^ 2023-10-11T11:06:45.171 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id= 2023-10-11T11:06:45.171 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 179 = '' 2023-10-11T11:06:45.172 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T11:06:45.172 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.8 2023-10-11T11:06:45.972 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_31 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_31.status 2023-10-11T11:06:45.972 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:06:45.973 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:06:45.973 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_31 2023-10-11T11:06:45.973 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_31.status 2023-10-11T11:06:45.973 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_31 --format xml 2023-10-11T11:06:45.973 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:06:46.016 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T11:06:46.019 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id=179 2023-10-11T11:06:46.019 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 179 = 179 2023-10-11T11:06:46.019 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:06:46.019 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_snap_present cluster1 mirror image_31 snap 2023-10-11T11:06:46.019 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:06:46.020 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:06:46.020 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_31 2023-10-11T11:06:46.020 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap 2023-10-11T11:06:46.020 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T11:06:46.020 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-10-11T11:06:46.020 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 1 2023-10-11T11:06:47.019 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 info mirror/image_31@snap 2023-10-11T11:06:47.073 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:rbd image 'image_31': 2023-10-11T11:06:47.073 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: size 128 MiB in 32 objects 2023-10-11T11:06:47.074 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: order 22 (4 MiB objects) 2023-10-11T11:06:47.074 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: snapshot_count: 2 2023-10-11T11:06:47.074 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: id: 12896dd74f4a 2023-10-11T11:06:47.074 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: block_name_prefix: rbd_data.12896dd74f4a 2023-10-11T11:06:47.074 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: format: 2 2023-10-11T11:06:47.075 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-10-11T11:06:47.075 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: op_features: 2023-10-11T11:06:47.075 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: flags: 2023-10-11T11:06:47.075 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: create_timestamp: Wed Oct 11 10:59:13 2023 2023-10-11T11:06:47.075 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: access_timestamp: Wed Oct 11 10:59:13 2023 2023-10-11T11:06:47.075 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: modify_timestamp: Wed Oct 11 10:59:13 2023 2023-10-11T11:06:47.076 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: protected: False 2023-10-11T11:06:47.076 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring state: enabled 2023-10-11T11:06:47.076 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring mode: snapshot 2023-10-11T11:06:47.076 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring global id: da5623f9-4467-416e-af1d-0ee95e25c12e 2023-10-11T11:06:47.076 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring primary: false 2023-10-11T11:06:47.078 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:06:47.078 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ compare_image_snaps mirror image_31 snap 2023-10-11T11:06:47.078 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:06:47.078 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_31 2023-10-11T11:06:47.078 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap 2023-10-11T11:06:47.079 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local ret=0 2023-10-11T11:06:47.079 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local rmt_export=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_31.export 2023-10-11T11:06:47.079 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local loc_export=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_31.export 2023-10-11T11:06:47.079 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rm -f /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_31.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_31.export 2023-10-11T11:06:47.079 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 -p mirror export image_31@snap /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_31.export 2023-10-11T11:06:47.265 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-10-11T11:06:47.271 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 -p mirror export image_31@snap /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_31.export 2023-10-11T11:06:47.511 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-10-11T11:06:47.518 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ cmp /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_31.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_31.export 2023-10-11T11:06:47.572 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rm -f /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_31.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_31.export 2023-10-11T11:06:47.608 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:06:47.609 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T11:06:47.609 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_32 2023-10-11T11:06:47.609 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ create_snap cluster2 mirror image_32 snap 2023-10-11T11:06:47.609 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:06:47.609 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:06:47.609 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_32 2023-10-11T11:06:47.610 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap 2023-10-11T11:06:47.610 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 -p mirror snap create image_32@snap --debug-rbd=20 --debug-journaler=20 2023-10-11T11:06:49.527 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_replay_started cluster1 mirror image_32 2023-10-11T11:06:49.527 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:06:49.527 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:06:49.528 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_32 2023-10-11T11:06:49.528 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_replay_state cluster1 mirror image_32 started 2023-10-11T11:06:49.528 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:06:49.528 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:06:49.528 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_32 2023-10-11T11:06:49.528 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local state=started 2023-10-11T11:06:49.529 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T11:06:49.529 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-10-11T11:06:49.529 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 1 2023-10-11T11:06:50.528 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test_image_replay_state cluster1 mirror image_32 started 2023-10-11T11:06:50.528 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:06:50.529 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:06:50.529 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_32 2023-10-11T11:06:50.529 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local test_state=started 2023-10-11T11:06:50.529 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local status_result 2023-10-11T11:06:50.529 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_state=stopped 2023-10-11T11:06:50.531 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_32 2023-10-11T11:06:50.531 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_instance=cluster1 2023-10-11T11:06:50.531 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ grep -i state 2023-10-11T11:06:50.531 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T11:06:50.531 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:06:50.532 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance=cluster1 2023-10-11T11:06:50.532 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local loop_instance 2023-10-11T11:06:50.532 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-10-11T11:06:50.532 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sleep 0 2023-10-11T11:06:50.532 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-10-11T11:06:50.532 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ seq 0 3 2023-10-11T11:06:50.534 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-10-11T11:06:50.534 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_32 2023-10-11T11:06:50.535 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1:0 2023-10-11T11:06:50.535 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T11:06:50.535 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance 2023-10-11T11:06:50.535 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-10-11T11:06:50.535 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local val=cluster1:0 2023-10-11T11:06:50.535 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_var_name=cluster 2023-10-11T11:06:50.536 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance_var_name=instance 2023-10-11T11:06:50.536 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ cluster=cluster1 2023-10-11T11:06:50.536 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ instance=0 2023-10-11T11:06:50.536 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' 0 = cluster1:0 ']' 2023-10-11T11:06:50.536 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval cluster=cluster1 2023-10-11T11:06:50.537 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:06:50.537 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval instance=0 2023-10-11T11:06:50.537 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=0 2023-10-11T11:06:50.537 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-10-11T11:06:50.537 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local local_cluster=cluster1:0 2023-10-11T11:06:50.537 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster=cluster1 2023-10-11T11:06:50.539 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance 2023-10-11T11:06:50.539 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-10-11T11:06:50.539 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local val=cluster1:0 2023-10-11T11:06:50.539 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster_var_name=local_cluster 2023-10-11T11:06:50.539 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance_var_name=instance 2023-10-11T11:06:50.540 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:06:50.540 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=0 2023-10-11T11:06:50.540 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ '[' 0 = cluster1:0 ']' 2023-10-11T11:06:50.540 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval local_cluster=cluster1 2023-10-11T11:06:50.540 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ local_cluster=cluster1 2023-10-11T11:06:50.541 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval instance=0 2023-10-11T11:06:50.541 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ instance=0 2023-10-11T11:06:50.541 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-10-11T11:06:50.555 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T11:06:50.555 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T11:06:50.555 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T11:06:50.555 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_32 2023-10-11T11:06:50.655 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:Invalid command: unused arguments: ['mirror/image_32'] 2023-10-11T11:06:50.656 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd mirror status : get status for rbd mirror 2023-10-11T11:06:50.656 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:admin_socket: invalid command 2023-10-11T11:06:50.659 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-10-11T11:06:50.659 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_32 2023-10-11T11:06:50.659 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1:1 2023-10-11T11:06:50.659 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T11:06:50.660 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance 2023-10-11T11:06:50.660 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-10-11T11:06:50.660 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local val=cluster1:1 2023-10-11T11:06:50.660 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_var_name=cluster 2023-10-11T11:06:50.660 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance_var_name=instance 2023-10-11T11:06:50.661 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ cluster=cluster1 2023-10-11T11:06:50.661 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ instance=1 2023-10-11T11:06:50.661 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' 1 = cluster1:1 ']' 2023-10-11T11:06:50.661 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval cluster=cluster1 2023-10-11T11:06:50.661 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:06:50.661 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval instance=1 2023-10-11T11:06:50.662 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=1 2023-10-11T11:06:50.662 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-10-11T11:06:50.662 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local local_cluster=cluster1:1 2023-10-11T11:06:50.662 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster=cluster1 2023-10-11T11:06:50.662 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance 2023-10-11T11:06:50.663 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-10-11T11:06:50.663 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local val=cluster1:1 2023-10-11T11:06:50.663 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster_var_name=local_cluster 2023-10-11T11:06:50.663 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance_var_name=instance 2023-10-11T11:06:50.663 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:06:50.663 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=1 2023-10-11T11:06:50.664 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ '[' 1 = cluster1:1 ']' 2023-10-11T11:06:50.664 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval local_cluster=cluster1 2023-10-11T11:06:50.664 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ local_cluster=cluster1 2023-10-11T11:06:50.664 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval instance=1 2023-10-11T11:06:50.664 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ instance=1 2023-10-11T11:06:50.664 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-10-11T11:06:50.678 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T11:06:50.678 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T11:06:50.678 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T11:06:50.678 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_32 2023-10-11T11:06:50.829 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ return 0 2023-10-11T11:06:50.830 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ status_result=' "state": "Replaying"' 2023-10-11T11:06:50.830 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ echo ' "state": "Replaying"' 2023-10-11T11:06:50.830 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ grep -i Replaying 2023-10-11T11:06:50.831 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: "state": "Replaying" 2023-10-11T11:06:50.832 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ current_state=started 2023-10-11T11:06:50.832 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test started = started 2023-10-11T11:06:50.833 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:06:50.833 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_32 2023-10-11T11:06:50.833 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local local_cluster=cluster1 2023-10-11T11:06:50.833 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:06:50.833 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:06:50.834 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_32 2023-10-11T11:06:50.834 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' snapshot = journal ']' 2023-10-11T11:06:50.834 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' snapshot = snapshot ']' 2023-10-11T11:06:50.834 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_32 2023-10-11T11:06:50.834 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local local_cluster=cluster1 2023-10-11T11:06:50.835 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:06:50.835 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:06:50.835 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_32 2023-10-11T11:06:50.835 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ mkfname cluster2-mirror-image_32.status 2023-10-11T11:06:50.835 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ echo cluster2-mirror-image_32.status 2023-10-11T11:06:50.835 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -e 's|[/ ]|_|g' 2023-10-11T11:06:50.837 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local status_log=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_32.status 2023-10-11T11:06:50.837 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ mkfname cluster1-mirror-image_32.status 2023-10-11T11:06:50.837 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ echo cluster1-mirror-image_32.status 2023-10-11T11:06:50.837 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -e 's|[/ ]|_|g' 2023-10-11T11:06:50.837 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local local_status_log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_32.status 2023-10-11T11:06:50.838 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ mirror_image_snapshot cluster2 mirror image_32 2023-10-11T11:06:50.838 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:06:50.838 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:06:50.838 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_32 2023-10-11T11:06:50.838 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_32 2023-10-11T11:06:51.526 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:Snapshot ID: 181 2023-10-11T11:06:51.538 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_32 /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_32.status 2023-10-11T11:06:51.538 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:06:51.538 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:06:51.538 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_32 2023-10-11T11:06:51.539 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_32.status 2023-10-11T11:06:51.539 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_32 --format xml 2023-10-11T11:06:51.539 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:06:51.581 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-10-11T11:06:51.583 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snapshot_id=181 2023-10-11T11:06:51.583 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T11:06:51.584 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T11:06:51.584 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.2 2023-10-11T11:06:51.784 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_32 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_32.status 2023-10-11T11:06:51.785 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:06:51.785 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:06:51.785 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_32 2023-10-11T11:06:51.785 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_32.status 2023-10-11T11:06:51.785 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_32 --format xml 2023-10-11T11:06:51.786 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:06:51.830 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T11:06:51.831 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T11:06:51.832 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:-:1.1: Document is empty 2023-10-11T11:06:51.832 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: 2023-10-11T11:06:51.832 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:^ 2023-10-11T11:06:51.833 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id= 2023-10-11T11:06:51.833 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 181 = '' 2023-10-11T11:06:51.833 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T11:06:51.834 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.4 2023-10-11T11:06:52.234 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_32 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_32.status 2023-10-11T11:06:52.234 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:06:52.234 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:06:52.235 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_32 2023-10-11T11:06:52.235 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_32.status 2023-10-11T11:06:52.235 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_32 --format xml 2023-10-11T11:06:52.235 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:06:52.283 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T11:06:52.283 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T11:06:52.285 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:-:1.1: Document is empty 2023-10-11T11:06:52.285 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: 2023-10-11T11:06:52.285 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:^ 2023-10-11T11:06:52.285 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id= 2023-10-11T11:06:52.286 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 181 = '' 2023-10-11T11:06:52.286 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T11:06:52.286 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.8 2023-10-11T11:06:53.087 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_32 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_32.status 2023-10-11T11:06:53.087 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:06:53.087 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:06:53.087 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_32 2023-10-11T11:06:53.088 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_32.status 2023-10-11T11:06:53.088 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_32 --format xml 2023-10-11T11:06:53.088 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:06:53.131 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T11:06:53.133 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id=181 2023-10-11T11:06:53.134 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 181 = 181 2023-10-11T11:06:53.134 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:06:53.134 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_snap_present cluster1 mirror image_32 snap 2023-10-11T11:06:53.134 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:06:53.134 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:06:53.135 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_32 2023-10-11T11:06:53.135 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap 2023-10-11T11:06:53.135 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T11:06:53.135 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-10-11T11:06:53.136 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 1 2023-10-11T11:06:54.134 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 info mirror/image_32@snap 2023-10-11T11:06:54.382 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:rbd image 'image_32': 2023-10-11T11:06:54.382 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: size 128 MiB in 32 objects 2023-10-11T11:06:54.383 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: order 22 (4 MiB objects) 2023-10-11T11:06:54.383 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: snapshot_count: 2 2023-10-11T11:06:54.383 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: id: 128941fae81 2023-10-11T11:06:54.383 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: block_name_prefix: rbd_data.128941fae81 2023-10-11T11:06:54.383 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: format: 2 2023-10-11T11:06:54.384 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-10-11T11:06:54.384 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: op_features: 2023-10-11T11:06:54.384 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: flags: 2023-10-11T11:06:54.384 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: create_timestamp: Wed Oct 11 10:59:23 2023 2023-10-11T11:06:54.384 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: access_timestamp: Wed Oct 11 10:59:23 2023 2023-10-11T11:06:54.384 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: modify_timestamp: Wed Oct 11 10:59:23 2023 2023-10-11T11:06:54.385 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: protected: False 2023-10-11T11:06:54.385 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring state: enabled 2023-10-11T11:06:54.385 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring mode: snapshot 2023-10-11T11:06:54.385 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring global id: b4fc3ca8-51a4-45f4-a987-97af54ee49dc 2023-10-11T11:06:54.385 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring primary: false 2023-10-11T11:06:54.386 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:06:54.387 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ compare_image_snaps mirror image_32 snap 2023-10-11T11:06:54.387 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:06:54.387 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_32 2023-10-11T11:06:54.387 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap 2023-10-11T11:06:54.387 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local ret=0 2023-10-11T11:06:54.388 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local rmt_export=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_32.export 2023-10-11T11:06:54.388 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local loc_export=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_32.export 2023-10-11T11:06:54.388 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rm -f /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_32.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_32.export 2023-10-11T11:06:54.388 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 -p mirror export image_32@snap /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_32.export 2023-10-11T11:06:54.565 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-10-11T11:06:54.571 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 -p mirror export image_32@snap /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_32.export 2023-10-11T11:06:54.823 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-10-11T11:06:54.831 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ cmp /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_32.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_32.export 2023-10-11T11:06:54.888 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rm -f /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_32.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_32.export 2023-10-11T11:06:54.925 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:06:54.925 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T11:06:54.926 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_33 2023-10-11T11:06:54.926 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ create_snap cluster2 mirror image_33 snap 2023-10-11T11:06:54.926 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:06:54.926 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:06:54.926 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_33 2023-10-11T11:06:54.926 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap 2023-10-11T11:06:54.927 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 -p mirror snap create image_33@snap --debug-rbd=20 --debug-journaler=20 2023-10-11T11:06:55.639 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.3777180363798974) > weight (0.25) 2023-10-11T11:06:55.639 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.9108165262735042) > weight (0.25) 2023-10-11T11:06:55.639 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.4874503449794456) > weight (0.25) 2023-10-11T11:06:55.640 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.5879480417380611) > weight (0.25) 2023-10-11T11:06:55.640 INFO:tasks.rbd_mirror_thrash:waiting for 92 secs before thrashing 2023-10-11T11:06:56.552 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_replay_started cluster1 mirror image_33 2023-10-11T11:06:56.553 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:06:56.553 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:06:56.553 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_33 2023-10-11T11:06:56.553 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_replay_state cluster1 mirror image_33 started 2023-10-11T11:06:56.554 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:06:56.554 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:06:56.554 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_33 2023-10-11T11:06:56.554 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local state=started 2023-10-11T11:06:56.554 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T11:06:56.555 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-10-11T11:06:56.555 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 1 2023-10-11T11:06:57.553 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test_image_replay_state cluster1 mirror image_33 started 2023-10-11T11:06:57.554 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:06:57.554 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:06:57.554 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_33 2023-10-11T11:06:57.554 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local test_state=started 2023-10-11T11:06:57.555 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local status_result 2023-10-11T11:06:57.555 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_state=stopped 2023-10-11T11:06:57.559 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_33 2023-10-11T11:06:57.560 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_instance=cluster1 2023-10-11T11:06:57.560 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T11:06:57.560 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:06:57.560 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ grep -i state 2023-10-11T11:06:57.560 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance=cluster1 2023-10-11T11:06:57.561 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local loop_instance 2023-10-11T11:06:57.561 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-10-11T11:06:57.561 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sleep 0 2023-10-11T11:06:57.561 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-10-11T11:06:57.562 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ seq 0 3 2023-10-11T11:06:57.562 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-10-11T11:06:57.562 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_33 2023-10-11T11:06:57.562 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1:0 2023-10-11T11:06:57.563 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T11:06:57.563 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance 2023-10-11T11:06:57.563 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-10-11T11:06:57.563 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local val=cluster1:0 2023-10-11T11:06:57.563 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_var_name=cluster 2023-10-11T11:06:57.564 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance_var_name=instance 2023-10-11T11:06:57.564 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ cluster=cluster1 2023-10-11T11:06:57.564 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ instance=0 2023-10-11T11:06:57.564 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' 0 = cluster1:0 ']' 2023-10-11T11:06:57.565 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval cluster=cluster1 2023-10-11T11:06:57.565 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:06:57.565 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval instance=0 2023-10-11T11:06:57.565 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=0 2023-10-11T11:06:57.565 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-10-11T11:06:57.566 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local local_cluster=cluster1:0 2023-10-11T11:06:57.566 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster=cluster1 2023-10-11T11:06:57.566 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance 2023-10-11T11:06:57.566 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-10-11T11:06:57.567 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local val=cluster1:0 2023-10-11T11:06:57.567 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster_var_name=local_cluster 2023-10-11T11:06:57.567 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance_var_name=instance 2023-10-11T11:06:57.567 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:06:57.568 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=0 2023-10-11T11:06:57.568 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ '[' 0 = cluster1:0 ']' 2023-10-11T11:06:57.568 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval local_cluster=cluster1 2023-10-11T11:06:57.568 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ local_cluster=cluster1 2023-10-11T11:06:57.568 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval instance=0 2023-10-11T11:06:57.569 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ instance=0 2023-10-11T11:06:57.569 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-10-11T11:06:57.580 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T11:06:57.581 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T11:06:57.581 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T11:06:57.581 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_33 2023-10-11T11:06:57.681 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:Invalid command: unused arguments: ['mirror/image_33'] 2023-10-11T11:06:57.682 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd mirror status : get status for rbd mirror 2023-10-11T11:06:57.682 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:admin_socket: invalid command 2023-10-11T11:06:57.685 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-10-11T11:06:57.685 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_33 2023-10-11T11:06:57.686 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1:1 2023-10-11T11:06:57.686 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T11:06:57.686 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance 2023-10-11T11:06:57.687 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-10-11T11:06:57.687 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local val=cluster1:1 2023-10-11T11:06:57.687 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_var_name=cluster 2023-10-11T11:06:57.687 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance_var_name=instance 2023-10-11T11:06:57.688 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ cluster=cluster1 2023-10-11T11:06:57.688 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ instance=1 2023-10-11T11:06:57.688 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' 1 = cluster1:1 ']' 2023-10-11T11:06:57.688 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval cluster=cluster1 2023-10-11T11:06:57.689 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:06:57.689 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval instance=1 2023-10-11T11:06:57.689 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=1 2023-10-11T11:06:57.689 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-10-11T11:06:57.690 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local local_cluster=cluster1:1 2023-10-11T11:06:57.690 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster=cluster1 2023-10-11T11:06:57.690 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance 2023-10-11T11:06:57.691 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-10-11T11:06:57.691 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local val=cluster1:1 2023-10-11T11:06:57.691 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster_var_name=local_cluster 2023-10-11T11:06:57.691 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance_var_name=instance 2023-10-11T11:06:57.692 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:06:57.692 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=1 2023-10-11T11:06:57.692 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ '[' 1 = cluster1:1 ']' 2023-10-11T11:06:57.692 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval local_cluster=cluster1 2023-10-11T11:06:57.693 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ local_cluster=cluster1 2023-10-11T11:06:57.693 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval instance=1 2023-10-11T11:06:57.693 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ instance=1 2023-10-11T11:06:57.693 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-10-11T11:06:57.704 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T11:06:57.704 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T11:06:57.704 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T11:06:57.705 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_33 2023-10-11T11:06:57.853 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ return 0 2023-10-11T11:06:57.853 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ status_result=' "state": "Replaying"' 2023-10-11T11:06:57.854 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ echo ' "state": "Replaying"' 2023-10-11T11:06:57.854 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ grep -i Replaying 2023-10-11T11:06:57.854 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: "state": "Replaying" 2023-10-11T11:06:57.857 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ current_state=started 2023-10-11T11:06:57.857 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test started = started 2023-10-11T11:06:57.858 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:06:57.858 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_33 2023-10-11T11:06:57.858 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local local_cluster=cluster1 2023-10-11T11:06:57.858 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:06:57.859 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:06:57.859 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_33 2023-10-11T11:06:57.859 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' snapshot = journal ']' 2023-10-11T11:06:57.859 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' snapshot = snapshot ']' 2023-10-11T11:06:57.860 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_33 2023-10-11T11:06:57.860 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local local_cluster=cluster1 2023-10-11T11:06:57.860 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:06:57.860 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:06:57.861 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_33 2023-10-11T11:06:57.861 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ mkfname cluster2-mirror-image_33.status 2023-10-11T11:06:57.861 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ echo cluster2-mirror-image_33.status 2023-10-11T11:06:57.862 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -e 's|[/ ]|_|g' 2023-10-11T11:06:57.862 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local status_log=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_33.status 2023-10-11T11:06:57.862 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ mkfname cluster1-mirror-image_33.status 2023-10-11T11:06:57.862 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ echo cluster1-mirror-image_33.status 2023-10-11T11:06:57.863 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -e 's|[/ ]|_|g' 2023-10-11T11:06:57.864 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local local_status_log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_33.status 2023-10-11T11:06:57.864 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ mirror_image_snapshot cluster2 mirror image_33 2023-10-11T11:06:57.864 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:06:57.864 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:06:57.865 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_33 2023-10-11T11:06:57.865 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_33 2023-10-11T11:06:58.557 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:Snapshot ID: 183 2023-10-11T11:06:58.568 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_33 /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_33.status 2023-10-11T11:06:58.568 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:06:58.568 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:06:58.568 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_33 2023-10-11T11:06:58.568 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_33.status 2023-10-11T11:06:58.568 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_33 --format xml 2023-10-11T11:06:58.569 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:06:58.610 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-10-11T11:06:58.612 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snapshot_id=183 2023-10-11T11:06:58.612 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T11:06:58.613 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T11:06:58.613 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.2 2023-10-11T11:06:58.814 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_33 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_33.status 2023-10-11T11:06:58.814 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:06:58.814 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:06:58.814 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_33 2023-10-11T11:06:58.814 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_33.status 2023-10-11T11:06:58.815 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_33 --format xml 2023-10-11T11:06:58.815 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:06:58.858 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T11:06:58.859 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T11:06:58.860 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:-:1.1: Document is empty 2023-10-11T11:06:58.861 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: 2023-10-11T11:06:58.861 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:^ 2023-10-11T11:06:58.861 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id= 2023-10-11T11:06:58.861 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 183 = '' 2023-10-11T11:06:58.861 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T11:06:58.861 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.4 2023-10-11T11:06:59.262 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_33 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_33.status 2023-10-11T11:06:59.262 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:06:59.262 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:06:59.263 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_33 2023-10-11T11:06:59.263 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_33.status 2023-10-11T11:06:59.263 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_33 --format xml 2023-10-11T11:06:59.263 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:06:59.309 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T11:06:59.309 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T11:06:59.310 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:-:1.1: Document is empty 2023-10-11T11:06:59.311 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: 2023-10-11T11:06:59.311 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:^ 2023-10-11T11:06:59.312 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id= 2023-10-11T11:06:59.312 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 183 = '' 2023-10-11T11:06:59.312 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T11:06:59.312 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.8 2023-10-11T11:07:00.112 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_33 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_33.status 2023-10-11T11:07:00.112 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:07:00.113 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:07:00.113 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_33 2023-10-11T11:07:00.113 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_33.status 2023-10-11T11:07:00.113 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_33 --format xml 2023-10-11T11:07:00.113 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:07:00.156 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T11:07:00.158 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id=183 2023-10-11T11:07:00.158 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 183 = 183 2023-10-11T11:07:00.158 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:07:00.159 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_snap_present cluster1 mirror image_33 snap 2023-10-11T11:07:00.159 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:07:00.159 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:07:00.159 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_33 2023-10-11T11:07:00.159 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap 2023-10-11T11:07:00.160 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T11:07:00.160 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-10-11T11:07:00.160 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 1 2023-10-11T11:07:01.159 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 info mirror/image_33@snap 2023-10-11T11:07:01.206 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:rbd image 'image_33': 2023-10-11T11:07:01.206 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: size 128 MiB in 32 objects 2023-10-11T11:07:01.206 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: order 22 (4 MiB objects) 2023-10-11T11:07:01.206 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: snapshot_count: 2 2023-10-11T11:07:01.207 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: id: 1289ed797b60 2023-10-11T11:07:01.207 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: block_name_prefix: rbd_data.1289ed797b60 2023-10-11T11:07:01.207 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: format: 2 2023-10-11T11:07:01.207 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-10-11T11:07:01.208 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: op_features: 2023-10-11T11:07:01.208 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: flags: 2023-10-11T11:07:01.208 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: create_timestamp: Wed Oct 11 10:59:29 2023 2023-10-11T11:07:01.208 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: access_timestamp: Wed Oct 11 10:59:29 2023 2023-10-11T11:07:01.208 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: modify_timestamp: Wed Oct 11 10:59:29 2023 2023-10-11T11:07:01.209 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: protected: False 2023-10-11T11:07:01.209 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring state: enabled 2023-10-11T11:07:01.209 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring mode: snapshot 2023-10-11T11:07:01.209 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring global id: a8456df7-6856-4049-9b99-fded34440c1f 2023-10-11T11:07:01.209 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring primary: false 2023-10-11T11:07:01.211 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:07:01.211 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ compare_image_snaps mirror image_33 snap 2023-10-11T11:07:01.211 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:07:01.211 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_33 2023-10-11T11:07:01.212 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap 2023-10-11T11:07:01.212 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local ret=0 2023-10-11T11:07:01.212 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local rmt_export=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_33.export 2023-10-11T11:07:01.212 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local loc_export=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_33.export 2023-10-11T11:07:01.212 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rm -f /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_33.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_33.export 2023-10-11T11:07:01.213 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 -p mirror export image_33@snap /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_33.export 2023-10-11T11:07:01.388 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-10-11T11:07:01.395 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 -p mirror export image_33@snap /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_33.export 2023-10-11T11:07:01.612 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-10-11T11:07:01.619 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ cmp /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_33.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_33.export 2023-10-11T11:07:01.673 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rm -f /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_33.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_33.export 2023-10-11T11:07:01.710 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:07:01.710 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T11:07:01.710 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_34 2023-10-11T11:07:01.711 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ create_snap cluster2 mirror image_34 snap 2023-10-11T11:07:01.711 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:07:01.711 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:07:01.711 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_34 2023-10-11T11:07:01.712 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap 2023-10-11T11:07:01.712 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 -p mirror snap create image_34@snap --debug-rbd=20 --debug-journaler=20 2023-10-11T11:07:03.571 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.5079638591235932) > weight (0.25) 2023-10-11T11:07:03.572 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.1 2023-10-11T11:07:03.572 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Sent signal 15 2023-10-11T11:07:03.572 INFO:tasks.rbd_mirror_thrash:waiting for 4 secs before reviving daemons 2023-10-11T11:07:03.573 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_replay_started cluster1 mirror image_34 2023-10-11T11:07:03.573 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:07:03.574 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:07:03.574 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_34 2023-10-11T11:07:03.574 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_replay_state cluster1 mirror image_34 started 2023-10-11T11:07:03.574 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:07:03.575 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:07:03.575 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_34 2023-10-11T11:07:03.575 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local state=started 2023-10-11T11:07:03.575 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T11:07:03.575 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-10-11T11:07:03.576 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 1 2023-10-11T11:07:03.576 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi111.stderr:2023-10-11T11:07:03.572+0000 7fd772464700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 114363) UID: 1000 2023-10-11T11:07:04.573 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test_image_replay_state cluster1 mirror image_34 started 2023-10-11T11:07:04.573 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:07:04.573 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:07:04.574 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_34 2023-10-11T11:07:04.574 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local test_state=started 2023-10-11T11:07:04.574 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local status_result 2023-10-11T11:07:04.574 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_state=stopped 2023-10-11T11:07:04.575 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_34 2023-10-11T11:07:04.575 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_instance=cluster1 2023-10-11T11:07:04.576 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ grep -i state 2023-10-11T11:07:04.576 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T11:07:04.576 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:07:04.576 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance=cluster1 2023-10-11T11:07:04.576 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local loop_instance 2023-10-11T11:07:04.577 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-10-11T11:07:04.577 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sleep 0 2023-10-11T11:07:04.577 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-10-11T11:07:04.577 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ seq 0 3 2023-10-11T11:07:04.579 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-10-11T11:07:04.579 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_34 2023-10-11T11:07:04.579 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1:0 2023-10-11T11:07:04.579 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T11:07:04.579 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance 2023-10-11T11:07:04.580 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-10-11T11:07:04.580 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local val=cluster1:0 2023-10-11T11:07:04.580 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_var_name=cluster 2023-10-11T11:07:04.580 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance_var_name=instance 2023-10-11T11:07:04.580 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ cluster=cluster1 2023-10-11T11:07:04.580 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ instance=0 2023-10-11T11:07:04.581 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' 0 = cluster1:0 ']' 2023-10-11T11:07:04.581 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval cluster=cluster1 2023-10-11T11:07:04.581 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:07:04.581 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval instance=0 2023-10-11T11:07:04.583 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=0 2023-10-11T11:07:04.583 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-10-11T11:07:04.583 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local local_cluster=cluster1:0 2023-10-11T11:07:04.583 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster=cluster1 2023-10-11T11:07:04.583 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance 2023-10-11T11:07:04.584 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-10-11T11:07:04.584 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local val=cluster1:0 2023-10-11T11:07:04.584 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster_var_name=local_cluster 2023-10-11T11:07:04.584 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance_var_name=instance 2023-10-11T11:07:04.584 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:07:04.584 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=0 2023-10-11T11:07:04.585 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ '[' 0 = cluster1:0 ']' 2023-10-11T11:07:04.585 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval local_cluster=cluster1 2023-10-11T11:07:04.585 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ local_cluster=cluster1 2023-10-11T11:07:04.585 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval instance=0 2023-10-11T11:07:04.585 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ instance=0 2023-10-11T11:07:04.586 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-10-11T11:07:04.599 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T11:07:04.600 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T11:07:04.600 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T11:07:04.600 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_34 2023-10-11T11:07:04.704 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:Invalid command: unused arguments: ['mirror/image_34'] 2023-10-11T11:07:04.705 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd mirror status : get status for rbd mirror 2023-10-11T11:07:04.705 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:admin_socket: invalid command 2023-10-11T11:07:04.707 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-10-11T11:07:04.707 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_34 2023-10-11T11:07:04.708 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1:1 2023-10-11T11:07:04.708 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T11:07:04.708 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance 2023-10-11T11:07:04.708 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-10-11T11:07:04.708 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local val=cluster1:1 2023-10-11T11:07:04.709 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_var_name=cluster 2023-10-11T11:07:04.709 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance_var_name=instance 2023-10-11T11:07:04.709 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ cluster=cluster1 2023-10-11T11:07:04.709 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ instance=1 2023-10-11T11:07:04.709 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' 1 = cluster1:1 ']' 2023-10-11T11:07:04.709 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval cluster=cluster1 2023-10-11T11:07:04.710 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:07:04.710 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval instance=1 2023-10-11T11:07:04.710 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=1 2023-10-11T11:07:04.710 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-10-11T11:07:04.710 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local local_cluster=cluster1:1 2023-10-11T11:07:04.710 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster=cluster1 2023-10-11T11:07:04.711 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance 2023-10-11T11:07:04.711 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-10-11T11:07:04.711 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local val=cluster1:1 2023-10-11T11:07:04.711 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster_var_name=local_cluster 2023-10-11T11:07:04.711 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance_var_name=instance 2023-10-11T11:07:04.711 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:07:04.712 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=1 2023-10-11T11:07:04.712 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ '[' 1 = cluster1:1 ']' 2023-10-11T11:07:04.712 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval local_cluster=cluster1 2023-10-11T11:07:04.712 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ local_cluster=cluster1 2023-10-11T11:07:04.712 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval instance=1 2023-10-11T11:07:04.712 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ instance=1 2023-10-11T11:07:04.713 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-10-11T11:07:04.726 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T11:07:04.727 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T11:07:04.727 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T11:07:04.727 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_34 2023-10-11T11:07:04.877 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ return 0 2023-10-11T11:07:04.878 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ status_result=' "state": "Replaying"' 2023-10-11T11:07:04.878 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ echo ' "state": "Replaying"' 2023-10-11T11:07:04.879 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ grep -i Replaying 2023-10-11T11:07:04.879 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: "state": "Replaying" 2023-10-11T11:07:04.881 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ current_state=started 2023-10-11T11:07:04.881 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test started = started 2023-10-11T11:07:04.881 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:07:04.881 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_34 2023-10-11T11:07:04.882 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local local_cluster=cluster1 2023-10-11T11:07:04.882 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:07:04.882 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:07:04.882 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_34 2023-10-11T11:07:04.882 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' snapshot = journal ']' 2023-10-11T11:07:04.883 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' snapshot = snapshot ']' 2023-10-11T11:07:04.883 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_34 2023-10-11T11:07:04.883 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local local_cluster=cluster1 2023-10-11T11:07:04.883 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:07:04.883 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:07:04.883 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_34 2023-10-11T11:07:04.884 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ mkfname cluster2-mirror-image_34.status 2023-10-11T11:07:04.884 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ echo cluster2-mirror-image_34.status 2023-10-11T11:07:04.884 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -e 's|[/ ]|_|g' 2023-10-11T11:07:04.884 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local status_log=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_34.status 2023-10-11T11:07:04.885 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ mkfname cluster1-mirror-image_34.status 2023-10-11T11:07:04.885 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ echo cluster1-mirror-image_34.status 2023-10-11T11:07:04.886 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -e 's|[/ ]|_|g' 2023-10-11T11:07:04.886 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local local_status_log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_34.status 2023-10-11T11:07:04.886 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ mirror_image_snapshot cluster2 mirror image_34 2023-10-11T11:07:04.886 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:07:04.886 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:07:04.887 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_34 2023-10-11T11:07:04.887 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_34 2023-10-11T11:07:05.576 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:Snapshot ID: 185 2023-10-11T11:07:05.586 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_34 /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_34.status 2023-10-11T11:07:05.587 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:07:05.587 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:07:05.587 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_34 2023-10-11T11:07:05.587 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_34.status 2023-10-11T11:07:05.587 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_34 --format xml 2023-10-11T11:07:05.588 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:07:05.635 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-10-11T11:07:05.637 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snapshot_id=185 2023-10-11T11:07:05.637 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T11:07:05.637 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T11:07:05.637 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.2 2023-10-11T11:07:05.838 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_34 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_34.status 2023-10-11T11:07:05.838 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:07:05.839 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:07:05.839 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_34 2023-10-11T11:07:05.839 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_34.status 2023-10-11T11:07:05.839 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_34 --format xml 2023-10-11T11:07:05.839 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:07:05.885 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T11:07:05.887 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id=101 2023-10-11T11:07:05.887 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 185 = 101 2023-10-11T11:07:05.887 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T11:07:05.887 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.4 2023-10-11T11:07:06.288 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_34 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_34.status 2023-10-11T11:07:06.291 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:07:06.291 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:07:06.291 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_34 2023-10-11T11:07:06.291 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_34.status 2023-10-11T11:07:06.291 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_34 --format xml 2023-10-11T11:07:06.292 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:07:06.336 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T11:07:06.337 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T11:07:06.338 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:-:1.1: Document is empty 2023-10-11T11:07:06.338 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: 2023-10-11T11:07:06.339 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:^ 2023-10-11T11:07:06.339 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id= 2023-10-11T11:07:06.339 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 185 = '' 2023-10-11T11:07:06.340 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T11:07:06.340 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.8 2023-10-11T11:07:07.140 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_34 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_34.status 2023-10-11T11:07:07.141 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:07:07.141 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:07:07.141 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_34 2023-10-11T11:07:07.141 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_34.status 2023-10-11T11:07:07.141 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_34 --format xml 2023-10-11T11:07:07.142 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:07:07.198 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T11:07:07.200 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id=185 2023-10-11T11:07:07.200 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 185 = 185 2023-10-11T11:07:07.201 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:07:07.201 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_snap_present cluster1 mirror image_34 snap 2023-10-11T11:07:07.201 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:07:07.201 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:07:07.201 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_34 2023-10-11T11:07:07.202 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap 2023-10-11T11:07:07.202 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T11:07:07.202 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-10-11T11:07:07.202 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 1 2023-10-11T11:07:07.573 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.1 2023-10-11T11:07:07.574 INFO:teuthology.orchestra.run:waiting for 600 2023-10-11T11:07:07.574 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.1 2023-10-11T11:07:07.575 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2023-10-11T11:07:07.575 DEBUG:teuthology.orchestra.run.smithi111:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2023-10-11T11:07:07.578 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2023-10-11T11:07:07.579 INFO:tasks.rbd_mirror_thrash:waiting for 111 secs before thrashing 2023-10-11T11:07:08.200 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 info mirror/image_34@snap 2023-10-11T11:07:08.241 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:rbd image 'image_34': 2023-10-11T11:07:08.241 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: size 128 MiB in 32 objects 2023-10-11T11:07:08.241 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: order 22 (4 MiB objects) 2023-10-11T11:07:08.241 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: snapshot_count: 2 2023-10-11T11:07:08.242 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: id: 1289568173fc 2023-10-11T11:07:08.242 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: block_name_prefix: rbd_data.1289568173fc 2023-10-11T11:07:08.242 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: format: 2 2023-10-11T11:07:08.242 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-10-11T11:07:08.243 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: op_features: 2023-10-11T11:07:08.243 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: flags: 2023-10-11T11:07:08.243 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: create_timestamp: Wed Oct 11 10:59:36 2023 2023-10-11T11:07:08.243 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: access_timestamp: Wed Oct 11 10:59:36 2023 2023-10-11T11:07:08.243 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: modify_timestamp: Wed Oct 11 10:59:36 2023 2023-10-11T11:07:08.244 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: protected: False 2023-10-11T11:07:08.244 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring state: enabled 2023-10-11T11:07:08.244 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring mode: snapshot 2023-10-11T11:07:08.244 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring global id: 78a70dcb-6515-4835-bfd0-d59363f95aba 2023-10-11T11:07:08.244 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring primary: false 2023-10-11T11:07:08.246 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:07:08.246 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ compare_image_snaps mirror image_34 snap 2023-10-11T11:07:08.246 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:07:08.246 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_34 2023-10-11T11:07:08.247 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap 2023-10-11T11:07:08.247 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local ret=0 2023-10-11T11:07:08.247 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local rmt_export=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_34.export 2023-10-11T11:07:08.247 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local loc_export=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_34.export 2023-10-11T11:07:08.248 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rm -f /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_34.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_34.export 2023-10-11T11:07:08.248 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 -p mirror export image_34@snap /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_34.export 2023-10-11T11:07:08.416 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-10-11T11:07:08.421 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 -p mirror export image_34@snap /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_34.export 2023-10-11T11:07:08.645 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-10-11T11:07:08.652 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ cmp /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_34.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_34.export 2023-10-11T11:07:08.708 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rm -f /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_34.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_34.export 2023-10-11T11:07:08.745 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:07:08.745 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T11:07:08.746 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_35 2023-10-11T11:07:08.746 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ create_snap cluster2 mirror image_35 snap 2023-10-11T11:07:08.746 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:07:08.746 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:07:08.746 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_35 2023-10-11T11:07:08.747 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap 2023-10-11T11:07:08.747 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 -p mirror snap create image_35@snap --debug-rbd=20 --debug-journaler=20 2023-10-11T11:07:10.590 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_replay_started cluster1 mirror image_35 2023-10-11T11:07:10.590 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:07:10.591 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:07:10.591 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_35 2023-10-11T11:07:10.591 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_replay_state cluster1 mirror image_35 started 2023-10-11T11:07:10.591 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:07:10.592 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:07:10.592 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_35 2023-10-11T11:07:10.592 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local state=started 2023-10-11T11:07:10.592 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T11:07:10.593 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-10-11T11:07:10.593 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 1 2023-10-11T11:07:11.591 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test_image_replay_state cluster1 mirror image_35 started 2023-10-11T11:07:11.591 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:07:11.591 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:07:11.591 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_35 2023-10-11T11:07:11.592 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local test_state=started 2023-10-11T11:07:11.592 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local status_result 2023-10-11T11:07:11.592 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_state=stopped 2023-10-11T11:07:11.593 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_35 2023-10-11T11:07:11.593 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ grep -i state 2023-10-11T11:07:11.594 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_instance=cluster1 2023-10-11T11:07:11.594 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T11:07:11.594 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:07:11.594 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance=cluster1 2023-10-11T11:07:11.594 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local loop_instance 2023-10-11T11:07:11.594 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-10-11T11:07:11.595 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sleep 0 2023-10-11T11:07:11.595 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-10-11T11:07:11.595 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ seq 0 3 2023-10-11T11:07:11.597 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-10-11T11:07:11.598 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_35 2023-10-11T11:07:11.598 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1:0 2023-10-11T11:07:11.598 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T11:07:11.598 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance 2023-10-11T11:07:11.598 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-10-11T11:07:11.599 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local val=cluster1:0 2023-10-11T11:07:11.599 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_var_name=cluster 2023-10-11T11:07:11.599 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance_var_name=instance 2023-10-11T11:07:11.599 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ cluster=cluster1 2023-10-11T11:07:11.599 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ instance=0 2023-10-11T11:07:11.600 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' 0 = cluster1:0 ']' 2023-10-11T11:07:11.600 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval cluster=cluster1 2023-10-11T11:07:11.600 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:07:11.600 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval instance=0 2023-10-11T11:07:11.600 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=0 2023-10-11T11:07:11.600 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-10-11T11:07:11.601 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local local_cluster=cluster1:0 2023-10-11T11:07:11.601 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster=cluster1 2023-10-11T11:07:11.601 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance 2023-10-11T11:07:11.601 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-10-11T11:07:11.601 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local val=cluster1:0 2023-10-11T11:07:11.602 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster_var_name=local_cluster 2023-10-11T11:07:11.602 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance_var_name=instance 2023-10-11T11:07:11.602 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:07:11.602 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=0 2023-10-11T11:07:11.602 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ '[' 0 = cluster1:0 ']' 2023-10-11T11:07:11.602 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval local_cluster=cluster1 2023-10-11T11:07:11.603 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ local_cluster=cluster1 2023-10-11T11:07:11.603 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval instance=0 2023-10-11T11:07:11.603 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ instance=0 2023-10-11T11:07:11.603 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-10-11T11:07:11.618 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T11:07:11.618 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T11:07:11.618 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T11:07:11.619 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_35 2023-10-11T11:07:11.719 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:Invalid command: unused arguments: ['mirror/image_35'] 2023-10-11T11:07:11.720 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd mirror status : get status for rbd mirror 2023-10-11T11:07:11.720 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:admin_socket: invalid command 2023-10-11T11:07:11.723 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-10-11T11:07:11.723 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_35 2023-10-11T11:07:11.723 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1:1 2023-10-11T11:07:11.724 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T11:07:11.724 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance 2023-10-11T11:07:11.724 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-10-11T11:07:11.724 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local val=cluster1:1 2023-10-11T11:07:11.724 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_var_name=cluster 2023-10-11T11:07:11.724 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance_var_name=instance 2023-10-11T11:07:11.725 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ cluster=cluster1 2023-10-11T11:07:11.725 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ instance=1 2023-10-11T11:07:11.725 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' 1 = cluster1:1 ']' 2023-10-11T11:07:11.725 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval cluster=cluster1 2023-10-11T11:07:11.725 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:07:11.726 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval instance=1 2023-10-11T11:07:11.726 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=1 2023-10-11T11:07:11.726 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-10-11T11:07:11.726 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local local_cluster=cluster1:1 2023-10-11T11:07:11.726 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster=cluster1 2023-10-11T11:07:11.727 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance 2023-10-11T11:07:11.727 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-10-11T11:07:11.727 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local val=cluster1:1 2023-10-11T11:07:11.727 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster_var_name=local_cluster 2023-10-11T11:07:11.727 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance_var_name=instance 2023-10-11T11:07:11.728 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:07:11.728 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=1 2023-10-11T11:07:11.728 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ '[' 1 = cluster1:1 ']' 2023-10-11T11:07:11.728 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval local_cluster=cluster1 2023-10-11T11:07:11.728 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ local_cluster=cluster1 2023-10-11T11:07:11.729 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval instance=1 2023-10-11T11:07:11.729 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ instance=1 2023-10-11T11:07:11.729 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-10-11T11:07:11.742 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T11:07:11.742 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T11:07:11.742 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T11:07:11.742 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_35 2023-10-11T11:07:11.892 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ return 0 2023-10-11T11:07:11.893 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ status_result=' "state": "Replaying"' 2023-10-11T11:07:11.893 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ echo ' "state": "Replaying"' 2023-10-11T11:07:11.894 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ grep -i Replaying 2023-10-11T11:07:11.894 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: "state": "Replaying" 2023-10-11T11:07:11.895 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ current_state=started 2023-10-11T11:07:11.896 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test started = started 2023-10-11T11:07:11.896 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:07:11.896 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_35 2023-10-11T11:07:11.896 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local local_cluster=cluster1 2023-10-11T11:07:11.896 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:07:11.897 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:07:11.897 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_35 2023-10-11T11:07:11.897 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' snapshot = journal ']' 2023-10-11T11:07:11.897 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' snapshot = snapshot ']' 2023-10-11T11:07:11.897 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_35 2023-10-11T11:07:11.897 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local local_cluster=cluster1 2023-10-11T11:07:11.898 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:07:11.898 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:07:11.898 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_35 2023-10-11T11:07:11.898 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ mkfname cluster2-mirror-image_35.status 2023-10-11T11:07:11.898 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ echo cluster2-mirror-image_35.status 2023-10-11T11:07:11.899 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -e 's|[/ ]|_|g' 2023-10-11T11:07:11.900 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local status_log=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_35.status 2023-10-11T11:07:11.900 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ mkfname cluster1-mirror-image_35.status 2023-10-11T11:07:11.900 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ echo cluster1-mirror-image_35.status 2023-10-11T11:07:11.900 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -e 's|[/ ]|_|g' 2023-10-11T11:07:11.900 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local local_status_log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_35.status 2023-10-11T11:07:11.901 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ mirror_image_snapshot cluster2 mirror image_35 2023-10-11T11:07:11.901 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:07:11.901 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:07:11.901 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_35 2023-10-11T11:07:11.901 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_35 2023-10-11T11:07:12.587 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:Snapshot ID: 187 2023-10-11T11:07:12.598 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_35 /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_35.status 2023-10-11T11:07:12.599 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:07:12.599 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:07:12.599 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_35 2023-10-11T11:07:12.599 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_35.status 2023-10-11T11:07:12.599 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_35 --format xml 2023-10-11T11:07:12.600 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:07:12.642 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-10-11T11:07:12.644 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snapshot_id=187 2023-10-11T11:07:12.644 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T11:07:12.644 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T11:07:12.644 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.2 2023-10-11T11:07:12.845 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_35 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_35.status 2023-10-11T11:07:12.846 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:07:12.846 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:07:12.846 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_35 2023-10-11T11:07:12.846 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_35.status 2023-10-11T11:07:12.846 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_35 --format xml 2023-10-11T11:07:12.847 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:07:12.893 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T11:07:12.895 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id=102 2023-10-11T11:07:12.895 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 187 = 102 2023-10-11T11:07:12.895 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T11:07:12.895 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.4 2023-10-11T11:07:13.296 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_35 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_35.status 2023-10-11T11:07:13.296 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:07:13.296 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:07:13.296 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_35 2023-10-11T11:07:13.297 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_35.status 2023-10-11T11:07:13.297 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_35 --format xml 2023-10-11T11:07:13.297 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:07:13.342 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T11:07:13.343 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T11:07:13.344 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:-:1.1: Document is empty 2023-10-11T11:07:13.345 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: 2023-10-11T11:07:13.345 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:^ 2023-10-11T11:07:13.345 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id= 2023-10-11T11:07:13.345 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 187 = '' 2023-10-11T11:07:13.346 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T11:07:13.346 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.8 2023-10-11T11:07:14.146 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_35 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_35.status 2023-10-11T11:07:14.146 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:07:14.146 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:07:14.146 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_35 2023-10-11T11:07:14.146 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_35.status 2023-10-11T11:07:14.147 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_35 --format xml 2023-10-11T11:07:14.147 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:07:14.200 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T11:07:14.200 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T11:07:14.202 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:-:1.1: Document is empty 2023-10-11T11:07:14.202 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: 2023-10-11T11:07:14.202 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:^ 2023-10-11T11:07:14.203 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id= 2023-10-11T11:07:14.203 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 187 = '' 2023-10-11T11:07:14.203 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T11:07:14.203 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 1.6 2023-10-11T11:07:15.804 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_35 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_35.status 2023-10-11T11:07:15.804 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:07:15.805 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:07:15.805 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_35 2023-10-11T11:07:15.805 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_35.status 2023-10-11T11:07:15.805 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_35 --format xml 2023-10-11T11:07:15.806 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:07:15.855 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T11:07:15.858 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id=187 2023-10-11T11:07:15.858 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 187 = 187 2023-10-11T11:07:15.858 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:07:15.859 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_snap_present cluster1 mirror image_35 snap 2023-10-11T11:07:15.859 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:07:15.859 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:07:15.859 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_35 2023-10-11T11:07:15.860 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap 2023-10-11T11:07:15.860 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T11:07:15.860 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-10-11T11:07:15.860 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 1 2023-10-11T11:07:16.858 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 info mirror/image_35@snap 2023-10-11T11:07:16.906 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:rbd image 'image_35': 2023-10-11T11:07:16.906 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: size 128 MiB in 32 objects 2023-10-11T11:07:16.906 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: order 22 (4 MiB objects) 2023-10-11T11:07:16.907 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: snapshot_count: 2 2023-10-11T11:07:16.907 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: id: 12893e5759b 2023-10-11T11:07:16.907 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: block_name_prefix: rbd_data.12893e5759b 2023-10-11T11:07:16.907 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: format: 2 2023-10-11T11:07:16.908 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-10-11T11:07:16.908 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: op_features: 2023-10-11T11:07:16.908 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: flags: 2023-10-11T11:07:16.908 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: create_timestamp: Wed Oct 11 10:59:49 2023 2023-10-11T11:07:16.908 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: access_timestamp: Wed Oct 11 10:59:49 2023 2023-10-11T11:07:16.909 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: modify_timestamp: Wed Oct 11 10:59:49 2023 2023-10-11T11:07:16.909 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: protected: False 2023-10-11T11:07:16.909 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring state: enabled 2023-10-11T11:07:16.909 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring mode: snapshot 2023-10-11T11:07:16.909 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring global id: d4ebf971-35de-4686-a8a6-e84c09701f07 2023-10-11T11:07:16.910 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring primary: false 2023-10-11T11:07:16.911 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:07:16.911 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ compare_image_snaps mirror image_35 snap 2023-10-11T11:07:16.911 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:07:16.911 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_35 2023-10-11T11:07:16.912 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap 2023-10-11T11:07:16.912 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local ret=0 2023-10-11T11:07:16.912 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local rmt_export=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_35.export 2023-10-11T11:07:16.912 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local loc_export=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_35.export 2023-10-11T11:07:16.912 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rm -f /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_35.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_35.export 2023-10-11T11:07:16.913 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 -p mirror export image_35@snap /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_35.export 2023-10-11T11:07:17.090 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-10-11T11:07:17.096 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 -p mirror export image_35@snap /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_35.export 2023-10-11T11:07:17.340 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-10-11T11:07:17.347 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ cmp /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_35.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_35.export 2023-10-11T11:07:17.401 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rm -f /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_35.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_35.export 2023-10-11T11:07:17.438 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:07:17.438 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T11:07:17.439 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_36 2023-10-11T11:07:17.439 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ create_snap cluster2 mirror image_36 snap 2023-10-11T11:07:17.439 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:07:17.439 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:07:17.440 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_36 2023-10-11T11:07:17.440 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap 2023-10-11T11:07:17.440 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 -p mirror snap create image_36@snap --debug-rbd=20 --debug-journaler=20 2023-10-11T11:07:18.610 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_replay_started cluster1 mirror image_36 2023-10-11T11:07:18.611 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:07:18.611 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:07:18.611 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_36 2023-10-11T11:07:18.612 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_replay_state cluster1 mirror image_36 started 2023-10-11T11:07:18.612 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:07:18.612 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:07:18.613 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_36 2023-10-11T11:07:18.613 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local state=started 2023-10-11T11:07:18.613 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T11:07:18.614 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-10-11T11:07:18.614 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 1 2023-10-11T11:07:19.611 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test_image_replay_state cluster1 mirror image_36 started 2023-10-11T11:07:19.611 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:07:19.611 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:07:19.612 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_36 2023-10-11T11:07:19.612 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local test_state=started 2023-10-11T11:07:19.612 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local status_result 2023-10-11T11:07:19.612 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_state=stopped 2023-10-11T11:07:19.613 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_36 2023-10-11T11:07:19.614 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ grep -i state 2023-10-11T11:07:19.614 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_instance=cluster1 2023-10-11T11:07:19.614 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T11:07:19.614 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:07:19.614 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance=cluster1 2023-10-11T11:07:19.615 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local loop_instance 2023-10-11T11:07:19.615 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-10-11T11:07:19.615 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sleep 0 2023-10-11T11:07:19.615 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-10-11T11:07:19.615 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ seq 0 3 2023-10-11T11:07:19.617 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-10-11T11:07:19.617 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_36 2023-10-11T11:07:19.617 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1:0 2023-10-11T11:07:19.617 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T11:07:19.618 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance 2023-10-11T11:07:19.618 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-10-11T11:07:19.618 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local val=cluster1:0 2023-10-11T11:07:19.618 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_var_name=cluster 2023-10-11T11:07:19.618 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance_var_name=instance 2023-10-11T11:07:19.618 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ cluster=cluster1 2023-10-11T11:07:19.619 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ instance=0 2023-10-11T11:07:19.619 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' 0 = cluster1:0 ']' 2023-10-11T11:07:19.619 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval cluster=cluster1 2023-10-11T11:07:19.619 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:07:19.619 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval instance=0 2023-10-11T11:07:19.620 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=0 2023-10-11T11:07:19.621 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-10-11T11:07:19.621 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local local_cluster=cluster1:0 2023-10-11T11:07:19.621 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster=cluster1 2023-10-11T11:07:19.622 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance 2023-10-11T11:07:19.622 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-10-11T11:07:19.622 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local val=cluster1:0 2023-10-11T11:07:19.622 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster_var_name=local_cluster 2023-10-11T11:07:19.622 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance_var_name=instance 2023-10-11T11:07:19.623 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:07:19.623 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=0 2023-10-11T11:07:19.623 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ '[' 0 = cluster1:0 ']' 2023-10-11T11:07:19.623 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval local_cluster=cluster1 2023-10-11T11:07:19.623 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ local_cluster=cluster1 2023-10-11T11:07:19.623 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval instance=0 2023-10-11T11:07:19.624 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ instance=0 2023-10-11T11:07:19.624 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-10-11T11:07:19.638 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T11:07:19.638 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T11:07:19.638 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T11:07:19.639 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_36 2023-10-11T11:07:19.741 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:Invalid command: unused arguments: ['mirror/image_36'] 2023-10-11T11:07:19.741 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd mirror status : get status for rbd mirror 2023-10-11T11:07:19.741 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:admin_socket: invalid command 2023-10-11T11:07:19.744 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-10-11T11:07:19.744 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_36 2023-10-11T11:07:19.745 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1:1 2023-10-11T11:07:19.745 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T11:07:19.745 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance 2023-10-11T11:07:19.745 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-10-11T11:07:19.745 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local val=cluster1:1 2023-10-11T11:07:19.745 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_var_name=cluster 2023-10-11T11:07:19.746 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance_var_name=instance 2023-10-11T11:07:19.746 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ cluster=cluster1 2023-10-11T11:07:19.746 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ instance=1 2023-10-11T11:07:19.746 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' 1 = cluster1:1 ']' 2023-10-11T11:07:19.746 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval cluster=cluster1 2023-10-11T11:07:19.747 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:07:19.747 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval instance=1 2023-10-11T11:07:19.747 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=1 2023-10-11T11:07:19.747 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-10-11T11:07:19.747 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local local_cluster=cluster1:1 2023-10-11T11:07:19.748 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster=cluster1 2023-10-11T11:07:19.748 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance 2023-10-11T11:07:19.748 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-10-11T11:07:19.748 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local val=cluster1:1 2023-10-11T11:07:19.748 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster_var_name=local_cluster 2023-10-11T11:07:19.748 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance_var_name=instance 2023-10-11T11:07:19.749 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:07:19.749 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=1 2023-10-11T11:07:19.749 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ '[' 1 = cluster1:1 ']' 2023-10-11T11:07:19.749 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval local_cluster=cluster1 2023-10-11T11:07:19.749 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ local_cluster=cluster1 2023-10-11T11:07:19.749 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval instance=1 2023-10-11T11:07:19.750 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ instance=1 2023-10-11T11:07:19.750 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-10-11T11:07:19.765 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T11:07:19.765 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T11:07:19.765 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T11:07:19.765 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_36 2023-10-11T11:07:19.918 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ return 0 2023-10-11T11:07:19.919 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ status_result=' "state": "Replaying"' 2023-10-11T11:07:19.919 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ echo ' "state": "Replaying"' 2023-10-11T11:07:19.919 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ grep -i Replaying 2023-10-11T11:07:19.920 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: "state": "Replaying" 2023-10-11T11:07:19.923 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ current_state=started 2023-10-11T11:07:19.923 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test started = started 2023-10-11T11:07:19.923 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:07:19.923 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_36 2023-10-11T11:07:19.923 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local local_cluster=cluster1 2023-10-11T11:07:19.924 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:07:19.924 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:07:19.924 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_36 2023-10-11T11:07:19.924 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' snapshot = journal ']' 2023-10-11T11:07:19.924 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' snapshot = snapshot ']' 2023-10-11T11:07:19.925 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_36 2023-10-11T11:07:19.925 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local local_cluster=cluster1 2023-10-11T11:07:19.925 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:07:19.926 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:07:19.926 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_36 2023-10-11T11:07:19.926 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ mkfname cluster2-mirror-image_36.status 2023-10-11T11:07:19.926 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ echo cluster2-mirror-image_36.status 2023-10-11T11:07:19.926 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -e 's|[/ ]|_|g' 2023-10-11T11:07:19.927 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local status_log=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_36.status 2023-10-11T11:07:19.927 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ mkfname cluster1-mirror-image_36.status 2023-10-11T11:07:19.927 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ echo cluster1-mirror-image_36.status 2023-10-11T11:07:19.927 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -e 's|[/ ]|_|g' 2023-10-11T11:07:19.929 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local local_status_log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_36.status 2023-10-11T11:07:19.929 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ mirror_image_snapshot cluster2 mirror image_36 2023-10-11T11:07:19.930 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:07:19.930 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:07:19.930 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_36 2023-10-11T11:07:19.930 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_36 2023-10-11T11:07:20.599 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:Snapshot ID: 189 2023-10-11T11:07:20.610 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_36 /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_36.status 2023-10-11T11:07:20.611 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:07:20.611 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:07:20.611 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_36 2023-10-11T11:07:20.611 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_36.status 2023-10-11T11:07:20.612 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_36 --format xml 2023-10-11T11:07:20.612 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:07:20.653 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-10-11T11:07:20.655 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snapshot_id=189 2023-10-11T11:07:20.655 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T11:07:20.655 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T11:07:20.655 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.2 2023-10-11T11:07:20.856 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_36 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_36.status 2023-10-11T11:07:20.856 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:07:20.857 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:07:20.857 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_36 2023-10-11T11:07:20.857 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_36.status 2023-10-11T11:07:20.857 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_36 --format xml 2023-10-11T11:07:20.857 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:07:20.901 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T11:07:20.901 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T11:07:20.903 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:-:1.1: Document is empty 2023-10-11T11:07:20.903 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: 2023-10-11T11:07:20.903 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:^ 2023-10-11T11:07:20.903 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id= 2023-10-11T11:07:20.904 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 189 = '' 2023-10-11T11:07:20.904 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T11:07:20.904 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.4 2023-10-11T11:07:21.304 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_36 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_36.status 2023-10-11T11:07:21.304 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:07:21.304 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:07:21.305 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_36 2023-10-11T11:07:21.305 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_36.status 2023-10-11T11:07:21.305 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_36 --format xml 2023-10-11T11:07:21.305 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:07:21.352 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T11:07:21.352 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T11:07:21.353 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:-:1.1: Document is empty 2023-10-11T11:07:21.353 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: 2023-10-11T11:07:21.353 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:^ 2023-10-11T11:07:21.354 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id= 2023-10-11T11:07:21.354 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 189 = '' 2023-10-11T11:07:21.355 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T11:07:21.355 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.8 2023-10-11T11:07:22.155 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_36 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_36.status 2023-10-11T11:07:22.155 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:07:22.156 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:07:22.156 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_36 2023-10-11T11:07:22.156 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_36.status 2023-10-11T11:07:22.156 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_36 --format xml 2023-10-11T11:07:22.156 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:07:22.201 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T11:07:22.203 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id=189 2023-10-11T11:07:22.203 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 189 = 189 2023-10-11T11:07:22.203 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:07:22.204 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_snap_present cluster1 mirror image_36 snap 2023-10-11T11:07:22.204 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:07:22.204 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:07:22.204 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_36 2023-10-11T11:07:22.204 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap 2023-10-11T11:07:22.205 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T11:07:22.205 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-10-11T11:07:22.205 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 1 2023-10-11T11:07:23.204 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 info mirror/image_36@snap 2023-10-11T11:07:23.250 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:rbd image 'image_36': 2023-10-11T11:07:23.250 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: size 128 MiB in 32 objects 2023-10-11T11:07:23.251 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: order 22 (4 MiB objects) 2023-10-11T11:07:23.251 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: snapshot_count: 2 2023-10-11T11:07:23.251 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: id: 128912e3a52c 2023-10-11T11:07:23.251 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: block_name_prefix: rbd_data.128912e3a52c 2023-10-11T11:07:23.251 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: format: 2 2023-10-11T11:07:23.251 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-10-11T11:07:23.252 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: op_features: 2023-10-11T11:07:23.252 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: flags: 2023-10-11T11:07:23.252 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: create_timestamp: Wed Oct 11 10:59:59 2023 2023-10-11T11:07:23.252 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: access_timestamp: Wed Oct 11 10:59:59 2023 2023-10-11T11:07:23.252 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: modify_timestamp: Wed Oct 11 10:59:59 2023 2023-10-11T11:07:23.253 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: protected: False 2023-10-11T11:07:23.253 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring state: enabled 2023-10-11T11:07:23.253 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring mode: snapshot 2023-10-11T11:07:23.253 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring global id: 26586d31-8ccc-4ee9-b1da-f7f6be6b7fa9 2023-10-11T11:07:23.253 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring primary: false 2023-10-11T11:07:23.254 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:07:23.255 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ compare_image_snaps mirror image_36 snap 2023-10-11T11:07:23.255 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:07:23.255 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_36 2023-10-11T11:07:23.255 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap 2023-10-11T11:07:23.255 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local ret=0 2023-10-11T11:07:23.255 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local rmt_export=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_36.export 2023-10-11T11:07:23.256 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local loc_export=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_36.export 2023-10-11T11:07:23.256 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rm -f /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_36.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_36.export 2023-10-11T11:07:23.256 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 -p mirror export image_36@snap /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_36.export 2023-10-11T11:07:23.440 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-10-11T11:07:23.446 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 -p mirror export image_36@snap /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_36.export 2023-10-11T11:07:23.677 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-10-11T11:07:23.684 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ cmp /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_36.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_36.export 2023-10-11T11:07:23.739 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rm -f /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_36.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_36.export 2023-10-11T11:07:23.775 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:07:23.776 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T11:07:23.776 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_37 2023-10-11T11:07:23.776 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ create_snap cluster2 mirror image_37 snap 2023-10-11T11:07:23.776 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:07:23.777 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:07:23.777 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_37 2023-10-11T11:07:23.777 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap 2023-10-11T11:07:23.777 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 -p mirror snap create image_37@snap --debug-rbd=20 --debug-journaler=20 2023-10-11T11:07:25.618 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_replay_started cluster1 mirror image_37 2023-10-11T11:07:25.618 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:07:25.618 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:07:25.618 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_37 2023-10-11T11:07:25.619 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_replay_state cluster1 mirror image_37 started 2023-10-11T11:07:25.619 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:07:25.619 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:07:25.619 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_37 2023-10-11T11:07:25.619 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local state=started 2023-10-11T11:07:25.619 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T11:07:25.620 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-10-11T11:07:25.620 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 1 2023-10-11T11:07:26.626 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test_image_replay_state cluster1 mirror image_37 started 2023-10-11T11:07:26.626 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:07:26.626 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:07:26.627 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_37 2023-10-11T11:07:26.627 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local test_state=started 2023-10-11T11:07:26.627 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local status_result 2023-10-11T11:07:26.627 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_state=stopped 2023-10-11T11:07:26.628 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_37 2023-10-11T11:07:26.628 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_instance=cluster1 2023-10-11T11:07:26.628 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ grep -i state 2023-10-11T11:07:26.629 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T11:07:26.629 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:07:26.629 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance=cluster1 2023-10-11T11:07:26.629 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local loop_instance 2023-10-11T11:07:26.630 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-10-11T11:07:26.630 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sleep 0 2023-10-11T11:07:26.630 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-10-11T11:07:26.630 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ seq 0 3 2023-10-11T11:07:26.631 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-10-11T11:07:26.631 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_37 2023-10-11T11:07:26.631 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1:0 2023-10-11T11:07:26.631 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T11:07:26.632 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance 2023-10-11T11:07:26.632 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-10-11T11:07:26.632 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local val=cluster1:0 2023-10-11T11:07:26.632 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_var_name=cluster 2023-10-11T11:07:26.633 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance_var_name=instance 2023-10-11T11:07:26.633 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ cluster=cluster1 2023-10-11T11:07:26.633 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ instance=0 2023-10-11T11:07:26.633 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' 0 = cluster1:0 ']' 2023-10-11T11:07:26.634 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval cluster=cluster1 2023-10-11T11:07:26.634 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:07:26.634 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval instance=0 2023-10-11T11:07:26.635 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=0 2023-10-11T11:07:26.635 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-10-11T11:07:26.635 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local local_cluster=cluster1:0 2023-10-11T11:07:26.635 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster=cluster1 2023-10-11T11:07:26.636 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance 2023-10-11T11:07:26.636 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-10-11T11:07:26.636 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local val=cluster1:0 2023-10-11T11:07:26.636 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster_var_name=local_cluster 2023-10-11T11:07:26.637 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance_var_name=instance 2023-10-11T11:07:26.637 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:07:26.637 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=0 2023-10-11T11:07:26.637 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ '[' 0 = cluster1:0 ']' 2023-10-11T11:07:26.638 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval local_cluster=cluster1 2023-10-11T11:07:26.638 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ local_cluster=cluster1 2023-10-11T11:07:26.638 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval instance=0 2023-10-11T11:07:26.638 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ instance=0 2023-10-11T11:07:26.639 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-10-11T11:07:26.647 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T11:07:26.647 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T11:07:26.647 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T11:07:26.648 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_37 2023-10-11T11:07:26.747 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:Invalid command: unused arguments: ['mirror/image_37'] 2023-10-11T11:07:26.748 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd mirror status : get status for rbd mirror 2023-10-11T11:07:26.748 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:admin_socket: invalid command 2023-10-11T11:07:26.751 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-10-11T11:07:26.752 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_37 2023-10-11T11:07:26.752 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1:1 2023-10-11T11:07:26.752 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T11:07:26.753 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance 2023-10-11T11:07:26.753 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-10-11T11:07:26.753 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local val=cluster1:1 2023-10-11T11:07:26.753 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_var_name=cluster 2023-10-11T11:07:26.754 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance_var_name=instance 2023-10-11T11:07:26.754 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ cluster=cluster1 2023-10-11T11:07:26.754 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ instance=1 2023-10-11T11:07:26.754 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' 1 = cluster1:1 ']' 2023-10-11T11:07:26.755 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval cluster=cluster1 2023-10-11T11:07:26.755 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:07:26.755 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval instance=1 2023-10-11T11:07:26.755 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=1 2023-10-11T11:07:26.756 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-10-11T11:07:26.756 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local local_cluster=cluster1:1 2023-10-11T11:07:26.756 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster=cluster1 2023-10-11T11:07:26.757 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance 2023-10-11T11:07:26.757 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-10-11T11:07:26.757 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local val=cluster1:1 2023-10-11T11:07:26.757 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster_var_name=local_cluster 2023-10-11T11:07:26.758 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance_var_name=instance 2023-10-11T11:07:26.758 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:07:26.758 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=1 2023-10-11T11:07:26.758 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ '[' 1 = cluster1:1 ']' 2023-10-11T11:07:26.759 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval local_cluster=cluster1 2023-10-11T11:07:26.759 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ local_cluster=cluster1 2023-10-11T11:07:26.759 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval instance=1 2023-10-11T11:07:26.759 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ instance=1 2023-10-11T11:07:26.760 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-10-11T11:07:26.770 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T11:07:26.770 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T11:07:26.770 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T11:07:26.771 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_37 2023-10-11T11:07:26.922 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ return 0 2023-10-11T11:07:26.923 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ status_result=' "state": "Replaying"' 2023-10-11T11:07:26.923 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ echo ' "state": "Replaying"' 2023-10-11T11:07:26.923 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ grep -i Replaying 2023-10-11T11:07:26.924 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: "state": "Replaying" 2023-10-11T11:07:26.926 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ current_state=started 2023-10-11T11:07:26.926 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test started = started 2023-10-11T11:07:26.926 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:07:26.927 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_37 2023-10-11T11:07:26.927 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local local_cluster=cluster1 2023-10-11T11:07:26.927 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:07:26.927 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:07:26.927 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_37 2023-10-11T11:07:26.928 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' snapshot = journal ']' 2023-10-11T11:07:26.928 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' snapshot = snapshot ']' 2023-10-11T11:07:26.928 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_37 2023-10-11T11:07:26.928 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local local_cluster=cluster1 2023-10-11T11:07:26.929 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:07:26.929 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:07:26.929 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_37 2023-10-11T11:07:26.929 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ mkfname cluster2-mirror-image_37.status 2023-10-11T11:07:26.929 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ echo cluster2-mirror-image_37.status 2023-10-11T11:07:26.930 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -e 's|[/ ]|_|g' 2023-10-11T11:07:26.930 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local status_log=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_37.status 2023-10-11T11:07:26.930 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ mkfname cluster1-mirror-image_37.status 2023-10-11T11:07:26.930 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ echo cluster1-mirror-image_37.status 2023-10-11T11:07:26.930 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -e 's|[/ ]|_|g' 2023-10-11T11:07:26.931 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local local_status_log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_37.status 2023-10-11T11:07:26.932 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ mirror_image_snapshot cluster2 mirror image_37 2023-10-11T11:07:26.932 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:07:26.932 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:07:26.932 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_37 2023-10-11T11:07:26.933 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_37 2023-10-11T11:07:27.622 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:Snapshot ID: 191 2023-10-11T11:07:27.634 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_37 /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_37.status 2023-10-11T11:07:27.634 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:07:27.635 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:07:27.635 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_37 2023-10-11T11:07:27.635 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_37.status 2023-10-11T11:07:27.635 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_37 --format xml 2023-10-11T11:07:27.635 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:07:27.677 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-10-11T11:07:27.679 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snapshot_id=191 2023-10-11T11:07:27.680 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T11:07:27.680 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T11:07:27.680 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.2 2023-10-11T11:07:27.881 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_37 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_37.status 2023-10-11T11:07:27.881 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:07:27.881 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:07:27.881 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_37 2023-10-11T11:07:27.881 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_37.status 2023-10-11T11:07:27.882 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_37 --format xml 2023-10-11T11:07:27.882 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:07:27.927 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T11:07:27.927 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T11:07:27.928 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:-:1.1: Document is empty 2023-10-11T11:07:27.928 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: 2023-10-11T11:07:27.929 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:^ 2023-10-11T11:07:27.929 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id= 2023-10-11T11:07:27.929 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 191 = '' 2023-10-11T11:07:27.930 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T11:07:27.930 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.4 2023-10-11T11:07:28.330 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_37 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_37.status 2023-10-11T11:07:28.331 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:07:28.331 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:07:28.331 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_37 2023-10-11T11:07:28.331 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_37.status 2023-10-11T11:07:28.332 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_37 --format xml 2023-10-11T11:07:28.332 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:07:28.377 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T11:07:28.378 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T11:07:28.379 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:-:1.1: Document is empty 2023-10-11T11:07:28.379 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: 2023-10-11T11:07:28.379 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:^ 2023-10-11T11:07:28.380 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id= 2023-10-11T11:07:28.380 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 191 = '' 2023-10-11T11:07:28.380 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T11:07:28.381 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.8 2023-10-11T11:07:29.181 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_37 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_37.status 2023-10-11T11:07:29.181 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:07:29.182 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:07:29.182 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_37 2023-10-11T11:07:29.182 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_37.status 2023-10-11T11:07:29.183 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_37 --format xml 2023-10-11T11:07:29.183 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:07:29.226 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T11:07:29.229 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id=191 2023-10-11T11:07:29.229 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 191 = 191 2023-10-11T11:07:29.229 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:07:29.230 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_snap_present cluster1 mirror image_37 snap 2023-10-11T11:07:29.230 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:07:29.230 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:07:29.231 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_37 2023-10-11T11:07:29.231 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap 2023-10-11T11:07:29.231 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T11:07:29.231 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-10-11T11:07:29.232 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 1 2023-10-11T11:07:30.229 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 info mirror/image_37@snap 2023-10-11T11:07:30.272 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:rbd image 'image_37': 2023-10-11T11:07:30.272 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: size 128 MiB in 32 objects 2023-10-11T11:07:30.272 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: order 22 (4 MiB objects) 2023-10-11T11:07:30.272 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: snapshot_count: 2 2023-10-11T11:07:30.272 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: id: 12897f720570 2023-10-11T11:07:30.273 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: block_name_prefix: rbd_data.12897f720570 2023-10-11T11:07:30.273 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: format: 2 2023-10-11T11:07:30.273 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-10-11T11:07:30.273 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: op_features: 2023-10-11T11:07:30.273 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: flags: 2023-10-11T11:07:30.273 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: create_timestamp: Wed Oct 11 11:00:05 2023 2023-10-11T11:07:30.274 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: access_timestamp: Wed Oct 11 11:00:05 2023 2023-10-11T11:07:30.274 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: modify_timestamp: Wed Oct 11 11:00:05 2023 2023-10-11T11:07:30.274 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: protected: False 2023-10-11T11:07:30.274 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring state: enabled 2023-10-11T11:07:30.274 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring mode: snapshot 2023-10-11T11:07:30.275 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring global id: 71f091e9-defc-4548-9e77-73505b063ba4 2023-10-11T11:07:30.275 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring primary: false 2023-10-11T11:07:30.276 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:07:30.276 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ compare_image_snaps mirror image_37 snap 2023-10-11T11:07:30.276 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:07:30.276 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_37 2023-10-11T11:07:30.277 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap 2023-10-11T11:07:30.277 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local ret=0 2023-10-11T11:07:30.277 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local rmt_export=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_37.export 2023-10-11T11:07:30.277 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local loc_export=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_37.export 2023-10-11T11:07:30.277 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rm -f /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_37.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_37.export 2023-10-11T11:07:30.278 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 -p mirror export image_37@snap /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_37.export 2023-10-11T11:07:30.448 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-10-11T11:07:30.454 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 -p mirror export image_37@snap /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_37.export 2023-10-11T11:07:30.716 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-10-11T11:07:30.723 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ cmp /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_37.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_37.export 2023-10-11T11:07:30.778 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rm -f /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_37.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_37.export 2023-10-11T11:07:30.815 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:07:30.815 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T11:07:30.815 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_38 2023-10-11T11:07:30.816 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ create_snap cluster2 mirror image_38 snap 2023-10-11T11:07:30.816 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:07:30.816 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:07:30.816 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_38 2023-10-11T11:07:30.816 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap 2023-10-11T11:07:30.817 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 -p mirror snap create image_38@snap --debug-rbd=20 --debug-journaler=20 2023-10-11T11:07:32.642 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_replay_started cluster1 mirror image_38 2023-10-11T11:07:32.643 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:07:32.643 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:07:32.643 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_38 2023-10-11T11:07:32.643 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_replay_state cluster1 mirror image_38 started 2023-10-11T11:07:32.643 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:07:32.643 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:07:32.644 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_38 2023-10-11T11:07:32.644 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local state=started 2023-10-11T11:07:32.644 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T11:07:32.644 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-10-11T11:07:32.644 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 1 2023-10-11T11:07:33.643 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test_image_replay_state cluster1 mirror image_38 started 2023-10-11T11:07:33.644 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:07:33.644 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:07:33.644 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_38 2023-10-11T11:07:33.644 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local test_state=started 2023-10-11T11:07:33.644 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local status_result 2023-10-11T11:07:33.645 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_state=stopped 2023-10-11T11:07:33.649 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_38 2023-10-11T11:07:33.649 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ grep -i state 2023-10-11T11:07:33.649 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_instance=cluster1 2023-10-11T11:07:33.649 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T11:07:33.649 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:07:33.650 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance=cluster1 2023-10-11T11:07:33.650 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local loop_instance 2023-10-11T11:07:33.650 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-10-11T11:07:33.650 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sleep 0 2023-10-11T11:07:33.650 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-10-11T11:07:33.651 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ seq 0 3 2023-10-11T11:07:33.651 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-10-11T11:07:33.651 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_38 2023-10-11T11:07:33.651 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1:0 2023-10-11T11:07:33.652 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T11:07:33.652 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance 2023-10-11T11:07:33.652 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-10-11T11:07:33.652 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local val=cluster1:0 2023-10-11T11:07:33.652 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_var_name=cluster 2023-10-11T11:07:33.653 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance_var_name=instance 2023-10-11T11:07:33.653 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ cluster=cluster1 2023-10-11T11:07:33.653 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ instance=0 2023-10-11T11:07:33.653 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' 0 = cluster1:0 ']' 2023-10-11T11:07:33.653 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval cluster=cluster1 2023-10-11T11:07:33.654 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:07:33.654 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval instance=0 2023-10-11T11:07:33.654 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=0 2023-10-11T11:07:33.654 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-10-11T11:07:33.654 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local local_cluster=cluster1:0 2023-10-11T11:07:33.655 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster=cluster1 2023-10-11T11:07:33.655 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance 2023-10-11T11:07:33.655 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-10-11T11:07:33.655 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local val=cluster1:0 2023-10-11T11:07:33.655 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster_var_name=local_cluster 2023-10-11T11:07:33.656 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance_var_name=instance 2023-10-11T11:07:33.656 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:07:33.656 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=0 2023-10-11T11:07:33.656 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ '[' 0 = cluster1:0 ']' 2023-10-11T11:07:33.656 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval local_cluster=cluster1 2023-10-11T11:07:33.657 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ local_cluster=cluster1 2023-10-11T11:07:33.657 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval instance=0 2023-10-11T11:07:33.657 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ instance=0 2023-10-11T11:07:33.657 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-10-11T11:07:33.670 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T11:07:33.671 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T11:07:33.671 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T11:07:33.671 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_38 2023-10-11T11:07:33.772 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:Invalid command: unused arguments: ['mirror/image_38'] 2023-10-11T11:07:33.773 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd mirror status : get status for rbd mirror 2023-10-11T11:07:33.773 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:admin_socket: invalid command 2023-10-11T11:07:33.776 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-10-11T11:07:33.776 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_38 2023-10-11T11:07:33.776 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1:1 2023-10-11T11:07:33.777 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T11:07:33.777 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance 2023-10-11T11:07:33.777 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-10-11T11:07:33.777 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local val=cluster1:1 2023-10-11T11:07:33.777 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_var_name=cluster 2023-10-11T11:07:33.778 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance_var_name=instance 2023-10-11T11:07:33.778 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ cluster=cluster1 2023-10-11T11:07:33.778 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ instance=1 2023-10-11T11:07:33.778 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' 1 = cluster1:1 ']' 2023-10-11T11:07:33.779 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval cluster=cluster1 2023-10-11T11:07:33.779 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:07:33.779 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval instance=1 2023-10-11T11:07:33.779 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=1 2023-10-11T11:07:33.779 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-10-11T11:07:33.780 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local local_cluster=cluster1:1 2023-10-11T11:07:33.780 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster=cluster1 2023-10-11T11:07:33.780 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance 2023-10-11T11:07:33.780 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-10-11T11:07:33.780 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local val=cluster1:1 2023-10-11T11:07:33.781 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster_var_name=local_cluster 2023-10-11T11:07:33.781 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance_var_name=instance 2023-10-11T11:07:33.781 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:07:33.781 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=1 2023-10-11T11:07:33.781 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ '[' 1 = cluster1:1 ']' 2023-10-11T11:07:33.782 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval local_cluster=cluster1 2023-10-11T11:07:33.782 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ local_cluster=cluster1 2023-10-11T11:07:33.782 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval instance=1 2023-10-11T11:07:33.782 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ instance=1 2023-10-11T11:07:33.782 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-10-11T11:07:33.795 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T11:07:33.795 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T11:07:33.795 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T11:07:33.796 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_38 2023-10-11T11:07:33.944 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ return 0 2023-10-11T11:07:33.945 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ status_result=' "state": "Replaying"' 2023-10-11T11:07:33.945 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ echo ' "state": "Replaying"' 2023-10-11T11:07:33.945 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ grep -i Replaying 2023-10-11T11:07:33.946 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: "state": "Replaying" 2023-10-11T11:07:33.947 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ current_state=started 2023-10-11T11:07:33.948 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test started = started 2023-10-11T11:07:33.948 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:07:33.948 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_38 2023-10-11T11:07:33.948 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local local_cluster=cluster1 2023-10-11T11:07:33.948 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:07:33.948 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:07:33.949 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_38 2023-10-11T11:07:33.949 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' snapshot = journal ']' 2023-10-11T11:07:33.949 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' snapshot = snapshot ']' 2023-10-11T11:07:33.949 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_38 2023-10-11T11:07:33.949 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local local_cluster=cluster1 2023-10-11T11:07:33.950 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:07:33.950 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:07:33.950 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_38 2023-10-11T11:07:33.950 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ mkfname cluster2-mirror-image_38.status 2023-10-11T11:07:33.950 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ echo cluster2-mirror-image_38.status 2023-10-11T11:07:33.950 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -e 's|[/ ]|_|g' 2023-10-11T11:07:33.951 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local status_log=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_38.status 2023-10-11T11:07:33.952 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ mkfname cluster1-mirror-image_38.status 2023-10-11T11:07:33.952 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ echo cluster1-mirror-image_38.status 2023-10-11T11:07:33.952 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -e 's|[/ ]|_|g' 2023-10-11T11:07:33.952 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local local_status_log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_38.status 2023-10-11T11:07:33.952 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ mirror_image_snapshot cluster2 mirror image_38 2023-10-11T11:07:33.953 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:07:33.953 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:07:33.953 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_38 2023-10-11T11:07:33.953 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_38 2023-10-11T11:07:34.650 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:Snapshot ID: 193 2023-10-11T11:07:34.661 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_38 /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_38.status 2023-10-11T11:07:34.661 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:07:34.661 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:07:34.661 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_38 2023-10-11T11:07:34.662 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_38.status 2023-10-11T11:07:34.662 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_38 --format xml 2023-10-11T11:07:34.662 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:07:34.704 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-10-11T11:07:34.706 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snapshot_id=193 2023-10-11T11:07:34.706 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T11:07:34.706 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T11:07:34.706 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.2 2023-10-11T11:07:34.907 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_38 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_38.status 2023-10-11T11:07:34.908 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:07:34.908 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:07:34.908 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_38 2023-10-11T11:07:34.908 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_38.status 2023-10-11T11:07:34.909 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_38 --format xml 2023-10-11T11:07:34.909 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:07:34.953 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T11:07:34.954 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T11:07:34.955 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:-:1.1: Document is empty 2023-10-11T11:07:34.955 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: 2023-10-11T11:07:34.955 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:^ 2023-10-11T11:07:34.956 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id= 2023-10-11T11:07:34.956 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 193 = '' 2023-10-11T11:07:34.956 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T11:07:34.957 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.4 2023-10-11T11:07:35.357 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_38 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_38.status 2023-10-11T11:07:35.357 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:07:35.357 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:07:35.358 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_38 2023-10-11T11:07:35.358 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_38.status 2023-10-11T11:07:35.358 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_38 --format xml 2023-10-11T11:07:35.358 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:07:35.403 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T11:07:35.404 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T11:07:35.404 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:-:1.1: Document is empty 2023-10-11T11:07:35.405 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: 2023-10-11T11:07:35.405 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:^ 2023-10-11T11:07:35.406 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id= 2023-10-11T11:07:35.406 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 193 = '' 2023-10-11T11:07:35.406 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T11:07:35.407 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.8 2023-10-11T11:07:36.207 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_38 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_38.status 2023-10-11T11:07:36.208 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:07:36.208 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:07:36.208 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_38 2023-10-11T11:07:36.208 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_38.status 2023-10-11T11:07:36.209 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_38 --format xml 2023-10-11T11:07:36.209 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:07:36.251 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T11:07:36.254 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id=193 2023-10-11T11:07:36.254 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 193 = 193 2023-10-11T11:07:36.255 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:07:36.255 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_snap_present cluster1 mirror image_38 snap 2023-10-11T11:07:36.255 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:07:36.255 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:07:36.256 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_38 2023-10-11T11:07:36.256 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap 2023-10-11T11:07:36.256 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T11:07:36.256 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-10-11T11:07:36.257 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 1 2023-10-11T11:07:37.257 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 info mirror/image_38@snap 2023-10-11T11:07:37.302 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:rbd image 'image_38': 2023-10-11T11:07:37.302 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: size 128 MiB in 32 objects 2023-10-11T11:07:37.302 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: order 22 (4 MiB objects) 2023-10-11T11:07:37.302 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: snapshot_count: 2 2023-10-11T11:07:37.302 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: id: 1289fd37009 2023-10-11T11:07:37.303 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: block_name_prefix: rbd_data.1289fd37009 2023-10-11T11:07:37.303 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: format: 2 2023-10-11T11:07:37.303 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-10-11T11:07:37.303 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: op_features: 2023-10-11T11:07:37.303 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: flags: 2023-10-11T11:07:37.304 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: create_timestamp: Wed Oct 11 11:00:11 2023 2023-10-11T11:07:37.304 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: access_timestamp: Wed Oct 11 11:00:11 2023 2023-10-11T11:07:37.304 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: modify_timestamp: Wed Oct 11 11:00:11 2023 2023-10-11T11:07:37.304 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: protected: False 2023-10-11T11:07:37.304 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring state: enabled 2023-10-11T11:07:37.304 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring mode: snapshot 2023-10-11T11:07:37.305 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring global id: 8f0a6f87-4fd6-4ab1-8d9b-69b67766f0e3 2023-10-11T11:07:37.305 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring primary: false 2023-10-11T11:07:37.306 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:07:37.306 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ compare_image_snaps mirror image_38 snap 2023-10-11T11:07:37.306 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:07:37.306 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_38 2023-10-11T11:07:37.307 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap 2023-10-11T11:07:37.307 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local ret=0 2023-10-11T11:07:37.307 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local rmt_export=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_38.export 2023-10-11T11:07:37.307 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local loc_export=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_38.export 2023-10-11T11:07:37.307 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rm -f /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_38.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_38.export 2023-10-11T11:07:37.308 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 -p mirror export image_38@snap /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_38.export 2023-10-11T11:07:37.486 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-10-11T11:07:37.493 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 -p mirror export image_38@snap /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_38.export 2023-10-11T11:07:37.721 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-10-11T11:07:37.726 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ cmp /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_38.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_38.export 2023-10-11T11:07:37.782 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rm -f /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_38.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_38.export 2023-10-11T11:07:37.820 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:07:37.820 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T11:07:37.820 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_39 2023-10-11T11:07:37.820 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ create_snap cluster2 mirror image_39 snap 2023-10-11T11:07:37.821 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:07:37.821 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:07:37.821 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_39 2023-10-11T11:07:37.821 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap 2023-10-11T11:07:37.822 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 -p mirror snap create image_39@snap --debug-rbd=20 --debug-journaler=20 2023-10-11T11:07:39.666 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_replay_started cluster1 mirror image_39 2023-10-11T11:07:39.667 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:07:39.667 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:07:39.668 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_39 2023-10-11T11:07:39.668 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_replay_state cluster1 mirror image_39 started 2023-10-11T11:07:39.668 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:07:39.668 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:07:39.669 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_39 2023-10-11T11:07:39.669 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local state=started 2023-10-11T11:07:39.669 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T11:07:39.669 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-10-11T11:07:39.670 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 1 2023-10-11T11:07:40.668 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test_image_replay_state cluster1 mirror image_39 started 2023-10-11T11:07:40.669 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:07:40.669 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:07:40.669 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_39 2023-10-11T11:07:40.669 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local test_state=started 2023-10-11T11:07:40.670 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local status_result 2023-10-11T11:07:40.670 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_state=stopped 2023-10-11T11:07:40.670 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_39 2023-10-11T11:07:40.670 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ grep -i state 2023-10-11T11:07:40.671 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_instance=cluster1 2023-10-11T11:07:40.671 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T11:07:40.671 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:07:40.672 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance=cluster1 2023-10-11T11:07:40.672 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local loop_instance 2023-10-11T11:07:40.672 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-10-11T11:07:40.672 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sleep 0 2023-10-11T11:07:40.676 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-10-11T11:07:40.676 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ seq 0 3 2023-10-11T11:07:40.676 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-10-11T11:07:40.677 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_39 2023-10-11T11:07:40.677 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1:0 2023-10-11T11:07:40.677 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T11:07:40.677 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance 2023-10-11T11:07:40.678 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-10-11T11:07:40.678 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local val=cluster1:0 2023-10-11T11:07:40.678 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_var_name=cluster 2023-10-11T11:07:40.679 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance_var_name=instance 2023-10-11T11:07:40.679 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ cluster=cluster1 2023-10-11T11:07:40.679 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ instance=0 2023-10-11T11:07:40.679 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' 0 = cluster1:0 ']' 2023-10-11T11:07:40.680 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval cluster=cluster1 2023-10-11T11:07:40.680 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:07:40.680 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval instance=0 2023-10-11T11:07:40.680 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=0 2023-10-11T11:07:40.681 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-10-11T11:07:40.681 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local local_cluster=cluster1:0 2023-10-11T11:07:40.681 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster=cluster1 2023-10-11T11:07:40.681 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance 2023-10-11T11:07:40.682 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-10-11T11:07:40.682 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local val=cluster1:0 2023-10-11T11:07:40.682 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster_var_name=local_cluster 2023-10-11T11:07:40.682 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance_var_name=instance 2023-10-11T11:07:40.683 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:07:40.683 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=0 2023-10-11T11:07:40.683 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ '[' 0 = cluster1:0 ']' 2023-10-11T11:07:40.683 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval local_cluster=cluster1 2023-10-11T11:07:40.684 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ local_cluster=cluster1 2023-10-11T11:07:40.684 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval instance=0 2023-10-11T11:07:40.684 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ instance=0 2023-10-11T11:07:40.684 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-10-11T11:07:40.696 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T11:07:40.696 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T11:07:40.697 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T11:07:40.697 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_39 2023-10-11T11:07:40.800 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:Invalid command: unused arguments: ['mirror/image_39'] 2023-10-11T11:07:40.800 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd mirror status : get status for rbd mirror 2023-10-11T11:07:40.801 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:admin_socket: invalid command 2023-10-11T11:07:40.804 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-10-11T11:07:40.804 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_39 2023-10-11T11:07:40.805 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1:1 2023-10-11T11:07:40.805 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T11:07:40.805 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance 2023-10-11T11:07:40.805 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-10-11T11:07:40.806 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local val=cluster1:1 2023-10-11T11:07:40.806 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_var_name=cluster 2023-10-11T11:07:40.806 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance_var_name=instance 2023-10-11T11:07:40.807 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ cluster=cluster1 2023-10-11T11:07:40.807 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ instance=1 2023-10-11T11:07:40.807 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' 1 = cluster1:1 ']' 2023-10-11T11:07:40.807 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval cluster=cluster1 2023-10-11T11:07:40.808 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:07:40.808 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval instance=1 2023-10-11T11:07:40.808 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=1 2023-10-11T11:07:40.808 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-10-11T11:07:40.809 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local local_cluster=cluster1:1 2023-10-11T11:07:40.809 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster=cluster1 2023-10-11T11:07:40.809 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance 2023-10-11T11:07:40.809 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-10-11T11:07:40.810 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local val=cluster1:1 2023-10-11T11:07:40.810 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster_var_name=local_cluster 2023-10-11T11:07:40.810 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance_var_name=instance 2023-10-11T11:07:40.811 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:07:40.811 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=1 2023-10-11T11:07:40.811 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ '[' 1 = cluster1:1 ']' 2023-10-11T11:07:40.811 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval local_cluster=cluster1 2023-10-11T11:07:40.812 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ local_cluster=cluster1 2023-10-11T11:07:40.812 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval instance=1 2023-10-11T11:07:40.812 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ instance=1 2023-10-11T11:07:40.812 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-10-11T11:07:40.823 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T11:07:40.823 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T11:07:40.824 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T11:07:40.824 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_39 2023-10-11T11:07:40.975 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ return 0 2023-10-11T11:07:40.976 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ status_result=' "state": "Replaying"' 2023-10-11T11:07:40.976 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ echo ' "state": "Replaying"' 2023-10-11T11:07:40.976 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ grep -i Replaying 2023-10-11T11:07:40.977 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: "state": "Replaying" 2023-10-11T11:07:40.980 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ current_state=started 2023-10-11T11:07:40.980 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test started = started 2023-10-11T11:07:40.981 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:07:40.981 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_39 2023-10-11T11:07:40.981 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local local_cluster=cluster1 2023-10-11T11:07:40.981 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:07:40.982 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:07:40.982 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_39 2023-10-11T11:07:40.982 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' snapshot = journal ']' 2023-10-11T11:07:40.982 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' snapshot = snapshot ']' 2023-10-11T11:07:40.983 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_39 2023-10-11T11:07:40.983 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local local_cluster=cluster1 2023-10-11T11:07:40.983 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:07:40.984 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:07:40.984 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_39 2023-10-11T11:07:40.984 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ mkfname cluster2-mirror-image_39.status 2023-10-11T11:07:40.984 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ echo cluster2-mirror-image_39.status 2023-10-11T11:07:40.985 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -e 's|[/ ]|_|g' 2023-10-11T11:07:40.985 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local status_log=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_39.status 2023-10-11T11:07:40.985 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ mkfname cluster1-mirror-image_39.status 2023-10-11T11:07:40.985 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ echo cluster1-mirror-image_39.status 2023-10-11T11:07:40.986 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -e 's|[/ ]|_|g' 2023-10-11T11:07:40.986 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local local_status_log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_39.status 2023-10-11T11:07:40.986 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ mirror_image_snapshot cluster2 mirror image_39 2023-10-11T11:07:40.987 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:07:40.987 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:07:40.987 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_39 2023-10-11T11:07:40.988 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_39 2023-10-11T11:07:41.662 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:Snapshot ID: 195 2023-10-11T11:07:41.674 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_39 /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_39.status 2023-10-11T11:07:41.674 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:07:41.675 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:07:41.675 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_39 2023-10-11T11:07:41.675 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_39.status 2023-10-11T11:07:41.676 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_39 --format xml 2023-10-11T11:07:41.676 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:07:41.720 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-10-11T11:07:41.722 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snapshot_id=195 2023-10-11T11:07:41.722 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T11:07:41.722 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T11:07:41.723 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.2 2023-10-11T11:07:41.923 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_39 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_39.status 2023-10-11T11:07:41.923 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:07:41.924 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:07:41.924 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_39 2023-10-11T11:07:41.925 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_39.status 2023-10-11T11:07:41.925 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_39 --format xml 2023-10-11T11:07:41.925 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:07:41.970 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T11:07:41.971 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T11:07:41.972 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:-:1.1: Document is empty 2023-10-11T11:07:41.972 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: 2023-10-11T11:07:41.973 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:^ 2023-10-11T11:07:41.973 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id= 2023-10-11T11:07:41.973 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 195 = '' 2023-10-11T11:07:41.973 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T11:07:41.974 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.4 2023-10-11T11:07:42.374 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_39 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_39.status 2023-10-11T11:07:42.374 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:07:42.374 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:07:42.374 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_39 2023-10-11T11:07:42.375 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_39.status 2023-10-11T11:07:42.375 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_39 --format xml 2023-10-11T11:07:42.375 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:07:42.626 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T11:07:42.627 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T11:07:42.629 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:-:1.1: Document is empty 2023-10-11T11:07:42.629 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: 2023-10-11T11:07:42.629 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:^ 2023-10-11T11:07:42.630 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id= 2023-10-11T11:07:42.630 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 195 = '' 2023-10-11T11:07:42.630 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T11:07:42.630 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.8 2023-10-11T11:07:42.774 INFO:tasks.ceph.cluster2.osd.1.smithi111.stderr:2023-10-11T11:07:42.768+0000 7f9628714700 -1 reset not still connected to 0x564242e32a90 2023-10-11T11:07:42.781 INFO:tasks.ceph.cluster2.osd.1.smithi111.stderr:2023-10-11T11:07:42.776+0000 7f9628714700 -1 reset not still connected to 0x56424a0672b0 2023-10-11T11:07:42.783 INFO:tasks.ceph.cluster2.osd.1.smithi111.stderr:2023-10-11T11:07:42.781+0000 7f9628714700 -1 reset not still connected to 0x56425695fba0 2023-10-11T11:07:42.783 INFO:tasks.ceph.cluster2.osd.1.smithi111.stderr:2023-10-11T11:07:42.781+0000 7f9628714700 -1 reset not still connected to 0x56425695fc70 2023-10-11T11:07:42.783 INFO:tasks.ceph.cluster2.osd.1.smithi111.stderr:2023-10-11T11:07:42.781+0000 7f9628714700 -1 reset not still connected to 0x56425695fd40 2023-10-11T11:07:42.783 INFO:tasks.ceph.cluster2.osd.1.smithi111.stderr:2023-10-11T11:07:42.781+0000 7f9628714700 -1 reset not still connected to 0x56425c95b450 2023-10-11T11:07:42.783 INFO:tasks.ceph.cluster2.osd.1.smithi111.stderr:2023-10-11T11:07:42.781+0000 7f9628714700 -1 reset not still connected to 0x5642603889c0 2023-10-11T11:07:42.784 INFO:tasks.ceph.cluster2.osd.1.smithi111.stderr:2023-10-11T11:07:42.781+0000 7f9628714700 -1 reset not still connected to 0x564264820c30 2023-10-11T11:07:42.784 INFO:tasks.ceph.cluster2.osd.1.smithi111.stderr:2023-10-11T11:07:42.781+0000 7f9628714700 -1 reset not still connected to 0x56426b4071e0 2023-10-11T11:07:42.784 INFO:tasks.ceph.cluster2.osd.1.smithi111.stderr:2023-10-11T11:07:42.781+0000 7f9628714700 -1 reset not still connected to 0x56426d4d80d0 2023-10-11T11:07:42.784 INFO:tasks.ceph.cluster2.osd.1.smithi111.stderr:2023-10-11T11:07:42.781+0000 7f9628714700 -1 reset not still connected to 0x564270ddba00 2023-10-11T11:07:42.784 INFO:tasks.ceph.cluster2.osd.1.smithi111.stderr:2023-10-11T11:07:42.781+0000 7f9628714700 -1 reset not still connected to 0x564277507860 2023-10-11T11:07:42.785 INFO:tasks.ceph.cluster2.osd.1.smithi111.stderr:2023-10-11T11:07:42.781+0000 7f9628714700 -1 reset not still connected to 0x56427c1efe10 2023-10-11T11:07:42.785 INFO:tasks.ceph.cluster2.osd.1.smithi111.stderr:2023-10-11T11:07:42.781+0000 7f9628714700 -1 reset not still connected to 0x56427f1d4000 2023-10-11T11:07:42.785 INFO:tasks.ceph.cluster2.osd.1.smithi111.stderr:2023-10-11T11:07:42.781+0000 7f9628714700 -1 reset not still connected to 0x56428c4ecb60 2023-10-11T11:07:43.431 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_39 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_39.status 2023-10-11T11:07:43.431 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:07:43.432 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:07:43.432 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_39 2023-10-11T11:07:43.432 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_39.status 2023-10-11T11:07:43.432 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_39 --format xml 2023-10-11T11:07:43.432 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:07:43.482 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T11:07:43.484 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id=195 2023-10-11T11:07:43.484 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 195 = 195 2023-10-11T11:07:43.484 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:07:43.484 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_snap_present cluster1 mirror image_39 snap 2023-10-11T11:07:43.485 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:07:43.485 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:07:43.485 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_39 2023-10-11T11:07:43.485 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap 2023-10-11T11:07:43.485 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T11:07:43.485 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-10-11T11:07:43.486 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 1 2023-10-11T11:07:44.485 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 info mirror/image_39@snap 2023-10-11T11:07:44.533 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:rbd image 'image_39': 2023-10-11T11:07:44.533 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: size 128 MiB in 32 objects 2023-10-11T11:07:44.533 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: order 22 (4 MiB objects) 2023-10-11T11:07:44.533 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: snapshot_count: 2 2023-10-11T11:07:44.533 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: id: 128954423842 2023-10-11T11:07:44.534 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: block_name_prefix: rbd_data.128954423842 2023-10-11T11:07:44.534 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: format: 2 2023-10-11T11:07:44.534 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-10-11T11:07:44.534 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: op_features: 2023-10-11T11:07:44.534 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: flags: 2023-10-11T11:07:44.534 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: create_timestamp: Wed Oct 11 11:00:24 2023 2023-10-11T11:07:44.535 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: access_timestamp: Wed Oct 11 11:00:24 2023 2023-10-11T11:07:44.535 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: modify_timestamp: Wed Oct 11 11:00:24 2023 2023-10-11T11:07:44.535 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: protected: False 2023-10-11T11:07:44.535 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring state: enabled 2023-10-11T11:07:44.535 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring mode: snapshot 2023-10-11T11:07:44.535 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring global id: 20fcc198-7ca6-4408-82dd-c95c583a202d 2023-10-11T11:07:44.536 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring primary: false 2023-10-11T11:07:44.537 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:07:44.537 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ compare_image_snaps mirror image_39 snap 2023-10-11T11:07:44.537 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:07:44.537 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_39 2023-10-11T11:07:44.537 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap 2023-10-11T11:07:44.538 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local ret=0 2023-10-11T11:07:44.538 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local rmt_export=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_39.export 2023-10-11T11:07:44.538 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local loc_export=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_39.export 2023-10-11T11:07:44.538 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rm -f /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_39.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_39.export 2023-10-11T11:07:44.539 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 -p mirror export image_39@snap /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_39.export 2023-10-11T11:07:44.716 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-10-11T11:07:44.722 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 -p mirror export image_39@snap /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_39.export 2023-10-11T11:07:44.960 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-10-11T11:07:44.966 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ cmp /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_39.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_39.export 2023-10-11T11:07:45.679 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rm -f /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_39.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_39.export 2023-10-11T11:07:45.717 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:07:45.717 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T11:07:45.717 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_40 2023-10-11T11:07:45.717 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ create_snap cluster2 mirror image_40 snap 2023-10-11T11:07:45.717 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:07:45.718 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:07:45.718 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_40 2023-10-11T11:07:45.718 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap 2023-10-11T11:07:45.718 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 -p mirror snap create image_40@snap --debug-rbd=20 --debug-journaler=20 2023-10-11T11:07:47.704 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_replay_started cluster1 mirror image_40 2023-10-11T11:07:47.704 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:07:47.704 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:07:47.705 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_40 2023-10-11T11:07:47.705 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_replay_state cluster1 mirror image_40 started 2023-10-11T11:07:47.705 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:07:47.705 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:07:47.705 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_40 2023-10-11T11:07:47.706 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local state=started 2023-10-11T11:07:47.706 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T11:07:47.706 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-10-11T11:07:47.706 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 1 2023-10-11T11:07:48.705 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test_image_replay_state cluster1 mirror image_40 started 2023-10-11T11:07:48.705 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:07:48.706 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:07:48.706 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_40 2023-10-11T11:07:48.706 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local test_state=started 2023-10-11T11:07:48.706 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local status_result 2023-10-11T11:07:48.706 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_state=stopped 2023-10-11T11:07:48.707 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_40 2023-10-11T11:07:48.708 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ grep -i state 2023-10-11T11:07:48.708 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_instance=cluster1 2023-10-11T11:07:48.708 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T11:07:48.708 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:07:48.708 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance=cluster1 2023-10-11T11:07:48.709 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local loop_instance 2023-10-11T11:07:48.709 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-10-11T11:07:48.709 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sleep 0 2023-10-11T11:07:48.709 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-10-11T11:07:48.709 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ seq 0 3 2023-10-11T11:07:48.711 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-10-11T11:07:48.711 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_40 2023-10-11T11:07:48.711 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1:0 2023-10-11T11:07:48.712 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T11:07:48.712 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance 2023-10-11T11:07:48.712 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-10-11T11:07:48.712 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local val=cluster1:0 2023-10-11T11:07:48.712 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_var_name=cluster 2023-10-11T11:07:48.713 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance_var_name=instance 2023-10-11T11:07:48.713 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ cluster=cluster1 2023-10-11T11:07:48.713 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ instance=0 2023-10-11T11:07:48.713 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' 0 = cluster1:0 ']' 2023-10-11T11:07:48.713 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval cluster=cluster1 2023-10-11T11:07:48.713 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:07:48.714 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval instance=0 2023-10-11T11:07:48.714 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=0 2023-10-11T11:07:48.714 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-10-11T11:07:48.714 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local local_cluster=cluster1:0 2023-10-11T11:07:48.714 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster=cluster1 2023-10-11T11:07:48.714 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance 2023-10-11T11:07:48.715 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-10-11T11:07:48.715 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local val=cluster1:0 2023-10-11T11:07:48.716 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster_var_name=local_cluster 2023-10-11T11:07:48.716 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance_var_name=instance 2023-10-11T11:07:48.716 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:07:48.716 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=0 2023-10-11T11:07:48.717 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ '[' 0 = cluster1:0 ']' 2023-10-11T11:07:48.717 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval local_cluster=cluster1 2023-10-11T11:07:48.717 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ local_cluster=cluster1 2023-10-11T11:07:48.717 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval instance=0 2023-10-11T11:07:48.717 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ instance=0 2023-10-11T11:07:48.718 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-10-11T11:07:48.732 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T11:07:48.733 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T11:07:48.733 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T11:07:48.733 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_40 2023-10-11T11:07:48.834 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:Invalid command: unused arguments: ['mirror/image_40'] 2023-10-11T11:07:48.834 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd mirror status : get status for rbd mirror 2023-10-11T11:07:48.834 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:admin_socket: invalid command 2023-10-11T11:07:48.837 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-10-11T11:07:48.837 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_40 2023-10-11T11:07:48.837 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1:1 2023-10-11T11:07:48.838 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T11:07:48.838 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance 2023-10-11T11:07:48.838 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-10-11T11:07:48.838 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local val=cluster1:1 2023-10-11T11:07:48.838 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_var_name=cluster 2023-10-11T11:07:48.839 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance_var_name=instance 2023-10-11T11:07:48.839 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ cluster=cluster1 2023-10-11T11:07:48.839 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ instance=1 2023-10-11T11:07:48.839 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' 1 = cluster1:1 ']' 2023-10-11T11:07:48.839 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval cluster=cluster1 2023-10-11T11:07:48.840 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:07:48.840 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval instance=1 2023-10-11T11:07:48.840 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=1 2023-10-11T11:07:48.840 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-10-11T11:07:48.840 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local local_cluster=cluster1:1 2023-10-11T11:07:48.840 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster=cluster1 2023-10-11T11:07:48.841 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance 2023-10-11T11:07:48.841 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-10-11T11:07:48.841 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local val=cluster1:1 2023-10-11T11:07:48.841 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster_var_name=local_cluster 2023-10-11T11:07:48.841 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance_var_name=instance 2023-10-11T11:07:48.841 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:07:48.842 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=1 2023-10-11T11:07:48.842 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ '[' 1 = cluster1:1 ']' 2023-10-11T11:07:48.842 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval local_cluster=cluster1 2023-10-11T11:07:48.842 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ local_cluster=cluster1 2023-10-11T11:07:48.842 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval instance=1 2023-10-11T11:07:48.843 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ instance=1 2023-10-11T11:07:48.843 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-10-11T11:07:48.857 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T11:07:48.858 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T11:07:48.858 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T11:07:48.858 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_40 2023-10-11T11:07:49.006 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ return 0 2023-10-11T11:07:49.007 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ status_result=' "state": "Replaying"' 2023-10-11T11:07:49.007 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ echo ' "state": "Replaying"' 2023-10-11T11:07:49.008 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ grep -i Replaying 2023-10-11T11:07:49.008 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: "state": "Replaying" 2023-10-11T11:07:49.010 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ current_state=started 2023-10-11T11:07:49.010 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test started = started 2023-10-11T11:07:49.010 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:07:49.010 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_40 2023-10-11T11:07:49.010 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local local_cluster=cluster1 2023-10-11T11:07:49.011 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:07:49.011 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:07:49.011 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_40 2023-10-11T11:07:49.011 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' snapshot = journal ']' 2023-10-11T11:07:49.011 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' snapshot = snapshot ']' 2023-10-11T11:07:49.012 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_40 2023-10-11T11:07:49.012 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local local_cluster=cluster1 2023-10-11T11:07:49.012 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:07:49.012 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:07:49.012 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_40 2023-10-11T11:07:49.012 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ mkfname cluster2-mirror-image_40.status 2023-10-11T11:07:49.013 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ echo cluster2-mirror-image_40.status 2023-10-11T11:07:49.013 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -e 's|[/ ]|_|g' 2023-10-11T11:07:49.013 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local status_log=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_40.status 2023-10-11T11:07:49.014 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ mkfname cluster1-mirror-image_40.status 2023-10-11T11:07:49.014 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ echo cluster1-mirror-image_40.status 2023-10-11T11:07:49.014 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -e 's|[/ ]|_|g' 2023-10-11T11:07:49.015 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local local_status_log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_40.status 2023-10-11T11:07:49.015 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ mirror_image_snapshot cluster2 mirror image_40 2023-10-11T11:07:49.015 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:07:49.015 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:07:49.015 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_40 2023-10-11T11:07:49.015 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_40 2023-10-11T11:07:49.708 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:Snapshot ID: 197 2023-10-11T11:07:49.719 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_40 /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_40.status 2023-10-11T11:07:49.719 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:07:49.719 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:07:49.720 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_40 2023-10-11T11:07:49.720 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_40.status 2023-10-11T11:07:49.720 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_40 --format xml 2023-10-11T11:07:49.721 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:07:49.767 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-10-11T11:07:49.769 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snapshot_id=197 2023-10-11T11:07:49.769 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T11:07:49.769 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T11:07:49.769 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.2 2023-10-11T11:07:49.970 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_40 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_40.status 2023-10-11T11:07:49.970 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:07:49.970 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:07:49.971 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_40 2023-10-11T11:07:49.971 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_40.status 2023-10-11T11:07:49.971 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_40 --format xml 2023-10-11T11:07:49.971 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:07:50.017 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T11:07:50.017 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T11:07:50.018 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:-:1.1: Document is empty 2023-10-11T11:07:50.018 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: 2023-10-11T11:07:50.019 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:^ 2023-10-11T11:07:50.019 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id= 2023-10-11T11:07:50.019 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 197 = '' 2023-10-11T11:07:50.020 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T11:07:50.020 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.4 2023-10-11T11:07:50.420 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_40 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_40.status 2023-10-11T11:07:50.420 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:07:50.420 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:07:50.421 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_40 2023-10-11T11:07:50.421 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_40.status 2023-10-11T11:07:50.421 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_40 --format xml 2023-10-11T11:07:50.421 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:07:50.468 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T11:07:50.469 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T11:07:50.470 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:-:1.1: Document is empty 2023-10-11T11:07:50.470 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: 2023-10-11T11:07:50.470 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:^ 2023-10-11T11:07:50.471 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id= 2023-10-11T11:07:50.471 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 197 = '' 2023-10-11T11:07:50.471 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T11:07:50.472 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.8 2023-10-11T11:07:51.272 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_40 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_40.status 2023-10-11T11:07:51.272 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:07:51.272 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:07:51.272 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_40 2023-10-11T11:07:51.272 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_40.status 2023-10-11T11:07:51.273 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_40 --format xml 2023-10-11T11:07:51.273 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:07:51.316 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T11:07:51.318 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id=197 2023-10-11T11:07:51.318 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 197 = 197 2023-10-11T11:07:51.318 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:07:51.319 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_snap_present cluster1 mirror image_40 snap 2023-10-11T11:07:51.319 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:07:51.319 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:07:51.319 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_40 2023-10-11T11:07:51.319 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap 2023-10-11T11:07:51.319 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T11:07:51.320 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-10-11T11:07:51.320 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 1 2023-10-11T11:07:52.319 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 info mirror/image_40@snap 2023-10-11T11:07:52.364 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:rbd image 'image_40': 2023-10-11T11:07:52.364 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: size 128 MiB in 32 objects 2023-10-11T11:07:52.365 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: order 22 (4 MiB objects) 2023-10-11T11:07:52.365 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: snapshot_count: 2 2023-10-11T11:07:52.365 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: id: 12896d3cc4fe 2023-10-11T11:07:52.365 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: block_name_prefix: rbd_data.12896d3cc4fe 2023-10-11T11:07:52.365 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: format: 2 2023-10-11T11:07:52.366 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-10-11T11:07:52.366 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: op_features: 2023-10-11T11:07:52.366 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: flags: 2023-10-11T11:07:52.366 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: create_timestamp: Wed Oct 11 11:00:31 2023 2023-10-11T11:07:52.366 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: access_timestamp: Wed Oct 11 11:00:31 2023 2023-10-11T11:07:52.367 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: modify_timestamp: Wed Oct 11 11:00:31 2023 2023-10-11T11:07:52.367 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: protected: False 2023-10-11T11:07:52.367 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring state: enabled 2023-10-11T11:07:52.367 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring mode: snapshot 2023-10-11T11:07:52.367 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring global id: 932fa6e9-efac-4d1a-a4b3-2f155e3296d4 2023-10-11T11:07:52.368 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring primary: false 2023-10-11T11:07:52.369 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:07:52.369 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ compare_image_snaps mirror image_40 snap 2023-10-11T11:07:52.369 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:07:52.369 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_40 2023-10-11T11:07:52.369 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap 2023-10-11T11:07:52.370 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local ret=0 2023-10-11T11:07:52.370 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local rmt_export=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_40.export 2023-10-11T11:07:52.370 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local loc_export=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_40.export 2023-10-11T11:07:52.370 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rm -f /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_40.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_40.export 2023-10-11T11:07:52.370 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 -p mirror export image_40@snap /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_40.export 2023-10-11T11:07:52.549 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-10-11T11:07:52.554 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 -p mirror export image_40@snap /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_40.export 2023-10-11T11:07:52.795 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-10-11T11:07:52.801 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ cmp /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_40.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_40.export 2023-10-11T11:07:52.856 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rm -f /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_40.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_40.export 2023-10-11T11:07:52.892 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:07:52.893 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T11:07:52.893 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_41 2023-10-11T11:07:52.893 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ create_snap cluster2 mirror image_41 snap 2023-10-11T11:07:52.893 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:07:52.894 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:07:52.894 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_41 2023-10-11T11:07:52.894 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap 2023-10-11T11:07:52.894 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 -p mirror snap create image_41@snap --debug-rbd=20 --debug-journaler=20 2023-10-11T11:07:54.717 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_replay_started cluster1 mirror image_41 2023-10-11T11:07:54.717 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:07:54.717 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:07:54.718 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_41 2023-10-11T11:07:54.718 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_replay_state cluster1 mirror image_41 started 2023-10-11T11:07:54.718 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:07:54.719 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:07:54.719 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_41 2023-10-11T11:07:54.719 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local state=started 2023-10-11T11:07:54.719 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T11:07:54.720 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-10-11T11:07:54.720 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 1 2023-10-11T11:07:55.717 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test_image_replay_state cluster1 mirror image_41 started 2023-10-11T11:07:55.718 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:07:55.718 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:07:55.718 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_41 2023-10-11T11:07:55.718 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local test_state=started 2023-10-11T11:07:55.718 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local status_result 2023-10-11T11:07:55.719 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_state=stopped 2023-10-11T11:07:55.720 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_41 2023-10-11T11:07:55.720 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_instance=cluster1 2023-10-11T11:07:55.720 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ grep -i state 2023-10-11T11:07:55.720 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T11:07:55.721 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:07:55.721 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance=cluster1 2023-10-11T11:07:55.721 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local loop_instance 2023-10-11T11:07:55.721 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-10-11T11:07:55.721 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sleep 0 2023-10-11T11:07:55.722 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-10-11T11:07:55.722 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ seq 0 3 2023-10-11T11:07:55.723 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-10-11T11:07:55.724 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_41 2023-10-11T11:07:55.724 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1:0 2023-10-11T11:07:55.724 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T11:07:55.724 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance 2023-10-11T11:07:55.724 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-10-11T11:07:55.725 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local val=cluster1:0 2023-10-11T11:07:55.725 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_var_name=cluster 2023-10-11T11:07:55.725 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance_var_name=instance 2023-10-11T11:07:55.725 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ cluster=cluster1 2023-10-11T11:07:55.725 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ instance=0 2023-10-11T11:07:55.726 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' 0 = cluster1:0 ']' 2023-10-11T11:07:55.726 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval cluster=cluster1 2023-10-11T11:07:55.726 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:07:55.726 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval instance=0 2023-10-11T11:07:55.726 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=0 2023-10-11T11:07:55.728 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-10-11T11:07:55.728 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local local_cluster=cluster1:0 2023-10-11T11:07:55.728 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster=cluster1 2023-10-11T11:07:55.728 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance 2023-10-11T11:07:55.729 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-10-11T11:07:55.729 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local val=cluster1:0 2023-10-11T11:07:55.729 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster_var_name=local_cluster 2023-10-11T11:07:55.729 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance_var_name=instance 2023-10-11T11:07:55.729 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:07:55.730 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=0 2023-10-11T11:07:55.730 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ '[' 0 = cluster1:0 ']' 2023-10-11T11:07:55.730 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval local_cluster=cluster1 2023-10-11T11:07:55.730 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ local_cluster=cluster1 2023-10-11T11:07:55.730 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval instance=0 2023-10-11T11:07:55.730 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ instance=0 2023-10-11T11:07:55.731 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-10-11T11:07:55.745 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T11:07:55.745 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T11:07:55.745 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T11:07:55.745 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_41 2023-10-11T11:07:55.848 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:Invalid command: unused arguments: ['mirror/image_41'] 2023-10-11T11:07:55.849 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd mirror status : get status for rbd mirror 2023-10-11T11:07:55.849 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:admin_socket: invalid command 2023-10-11T11:07:55.852 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-10-11T11:07:55.852 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_41 2023-10-11T11:07:55.852 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1:1 2023-10-11T11:07:55.853 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T11:07:55.853 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance 2023-10-11T11:07:55.853 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-10-11T11:07:55.853 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local val=cluster1:1 2023-10-11T11:07:55.854 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_var_name=cluster 2023-10-11T11:07:55.854 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance_var_name=instance 2023-10-11T11:07:55.854 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ cluster=cluster1 2023-10-11T11:07:55.854 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ instance=1 2023-10-11T11:07:55.854 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' 1 = cluster1:1 ']' 2023-10-11T11:07:55.855 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval cluster=cluster1 2023-10-11T11:07:55.855 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:07:55.855 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval instance=1 2023-10-11T11:07:55.855 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=1 2023-10-11T11:07:55.855 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-10-11T11:07:55.856 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local local_cluster=cluster1:1 2023-10-11T11:07:55.856 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster=cluster1 2023-10-11T11:07:55.856 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance 2023-10-11T11:07:55.856 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-10-11T11:07:55.856 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local val=cluster1:1 2023-10-11T11:07:55.857 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster_var_name=local_cluster 2023-10-11T11:07:55.857 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance_var_name=instance 2023-10-11T11:07:55.857 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:07:55.857 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=1 2023-10-11T11:07:55.857 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ '[' 1 = cluster1:1 ']' 2023-10-11T11:07:55.858 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval local_cluster=cluster1 2023-10-11T11:07:55.858 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ local_cluster=cluster1 2023-10-11T11:07:55.858 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval instance=1 2023-10-11T11:07:55.858 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ instance=1 2023-10-11T11:07:55.859 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-10-11T11:07:55.871 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T11:07:55.871 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T11:07:55.871 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T11:07:55.871 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_41 2023-10-11T11:07:56.021 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ return 0 2023-10-11T11:07:56.021 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ status_result=' "state": "Replaying"' 2023-10-11T11:07:56.021 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ echo ' "state": "Replaying"' 2023-10-11T11:07:56.022 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ grep -i Replaying 2023-10-11T11:07:56.022 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: "state": "Replaying" 2023-10-11T11:07:56.026 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ current_state=started 2023-10-11T11:07:56.026 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test started = started 2023-10-11T11:07:56.026 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:07:56.027 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_41 2023-10-11T11:07:56.027 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local local_cluster=cluster1 2023-10-11T11:07:56.027 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:07:56.028 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:07:56.028 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_41 2023-10-11T11:07:56.028 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' snapshot = journal ']' 2023-10-11T11:07:56.028 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' snapshot = snapshot ']' 2023-10-11T11:07:56.029 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_41 2023-10-11T11:07:56.029 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local local_cluster=cluster1 2023-10-11T11:07:56.029 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:07:56.029 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:07:56.030 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_41 2023-10-11T11:07:56.030 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ mkfname cluster2-mirror-image_41.status 2023-10-11T11:07:56.030 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ echo cluster2-mirror-image_41.status 2023-10-11T11:07:56.031 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -e 's|[/ ]|_|g' 2023-10-11T11:07:56.031 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local status_log=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_41.status 2023-10-11T11:07:56.031 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ mkfname cluster1-mirror-image_41.status 2023-10-11T11:07:56.031 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ echo cluster1-mirror-image_41.status 2023-10-11T11:07:56.032 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -e 's|[/ ]|_|g' 2023-10-11T11:07:56.032 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local local_status_log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_41.status 2023-10-11T11:07:56.032 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ mirror_image_snapshot cluster2 mirror image_41 2023-10-11T11:07:56.032 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:07:56.033 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:07:56.033 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_41 2023-10-11T11:07:56.033 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_41 2023-10-11T11:07:56.715 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:Snapshot ID: 199 2023-10-11T11:07:56.726 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_41 /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_41.status 2023-10-11T11:07:56.726 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:07:56.726 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:07:56.727 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_41 2023-10-11T11:07:56.727 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_41.status 2023-10-11T11:07:56.727 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_41 --format xml 2023-10-11T11:07:56.727 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:07:56.768 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-10-11T11:07:56.770 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snapshot_id=199 2023-10-11T11:07:56.770 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T11:07:56.771 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T11:07:56.771 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.2 2023-10-11T11:07:56.971 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_41 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_41.status 2023-10-11T11:07:56.972 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:07:56.972 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:07:56.972 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_41 2023-10-11T11:07:56.972 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_41.status 2023-10-11T11:07:56.972 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_41 --format xml 2023-10-11T11:07:56.973 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:07:57.019 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T11:07:57.021 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id=108 2023-10-11T11:07:57.021 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 199 = 108 2023-10-11T11:07:57.021 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T11:07:57.022 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.4 2023-10-11T11:07:57.422 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_41 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_41.status 2023-10-11T11:07:57.422 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:07:57.423 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:07:57.423 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_41 2023-10-11T11:07:57.423 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_41.status 2023-10-11T11:07:57.423 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_41 --format xml 2023-10-11T11:07:57.423 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:07:57.481 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T11:07:57.484 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id=108 2023-10-11T11:07:57.484 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 199 = 108 2023-10-11T11:07:57.484 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T11:07:57.484 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.8 2023-10-11T11:07:58.285 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_41 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_41.status 2023-10-11T11:07:58.285 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:07:58.285 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:07:58.286 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_41 2023-10-11T11:07:58.286 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_41.status 2023-10-11T11:07:58.286 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_41 --format xml 2023-10-11T11:07:58.286 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:07:58.338 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T11:07:58.338 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T11:07:58.340 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:-:1.1: Document is empty 2023-10-11T11:07:58.340 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: 2023-10-11T11:07:58.340 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:^ 2023-10-11T11:07:58.341 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id= 2023-10-11T11:07:58.341 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 199 = '' 2023-10-11T11:07:58.341 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T11:07:58.341 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 1.6 2023-10-11T11:07:59.942 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_41 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_41.status 2023-10-11T11:07:59.942 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:07:59.942 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:07:59.943 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_41 2023-10-11T11:07:59.943 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_41.status 2023-10-11T11:07:59.943 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_41 --format xml 2023-10-11T11:07:59.943 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:07:59.989 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T11:07:59.991 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id=199 2023-10-11T11:07:59.991 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 199 = 199 2023-10-11T11:07:59.992 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:07:59.992 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_snap_present cluster1 mirror image_41 snap 2023-10-11T11:07:59.992 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:07:59.992 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:07:59.992 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_41 2023-10-11T11:07:59.993 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap 2023-10-11T11:07:59.993 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T11:07:59.993 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-10-11T11:07:59.993 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 1 2023-10-11T11:08:00.992 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 info mirror/image_41@snap 2023-10-11T11:08:01.037 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:rbd image 'image_41': 2023-10-11T11:08:01.037 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: size 128 MiB in 32 objects 2023-10-11T11:08:01.037 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: order 22 (4 MiB objects) 2023-10-11T11:08:01.038 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: snapshot_count: 2 2023-10-11T11:08:01.038 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: id: 12895626c4aa 2023-10-11T11:08:01.038 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: block_name_prefix: rbd_data.12895626c4aa 2023-10-11T11:08:01.038 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: format: 2 2023-10-11T11:08:01.039 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-10-11T11:08:01.039 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: op_features: 2023-10-11T11:08:01.039 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: flags: 2023-10-11T11:08:01.040 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: create_timestamp: Wed Oct 11 11:00:41 2023 2023-10-11T11:08:01.040 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: access_timestamp: Wed Oct 11 11:00:41 2023 2023-10-11T11:08:01.040 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: modify_timestamp: Wed Oct 11 11:00:41 2023 2023-10-11T11:08:01.040 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: protected: False 2023-10-11T11:08:01.041 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring state: enabled 2023-10-11T11:08:01.041 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring mode: snapshot 2023-10-11T11:08:01.041 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring global id: 981fef91-675f-4aae-8d59-cba97f3e8943 2023-10-11T11:08:01.041 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring primary: false 2023-10-11T11:08:01.043 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:08:01.043 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ compare_image_snaps mirror image_41 snap 2023-10-11T11:08:01.044 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:08:01.044 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_41 2023-10-11T11:08:01.044 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap 2023-10-11T11:08:01.044 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local ret=0 2023-10-11T11:08:01.045 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local rmt_export=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_41.export 2023-10-11T11:08:01.045 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local loc_export=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_41.export 2023-10-11T11:08:01.045 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rm -f /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_41.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_41.export 2023-10-11T11:08:01.045 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 -p mirror export image_41@snap /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_41.export 2023-10-11T11:08:01.217 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-10-11T11:08:01.223 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 -p mirror export image_41@snap /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_41.export 2023-10-11T11:08:01.474 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-10-11T11:08:01.480 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ cmp /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_41.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_41.export 2023-10-11T11:08:01.536 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rm -f /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_41.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_41.export 2023-10-11T11:08:01.572 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:08:01.573 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T11:08:01.573 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_42 2023-10-11T11:08:01.573 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ create_snap cluster2 mirror image_42 snap 2023-10-11T11:08:01.574 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:08:01.574 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:08:01.574 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_42 2023-10-11T11:08:01.574 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap 2023-10-11T11:08:01.575 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 -p mirror snap create image_42@snap --debug-rbd=20 --debug-journaler=20 2023-10-11T11:08:03.426 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_replay_started cluster1 mirror image_42 2023-10-11T11:08:03.426 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:08:03.426 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:08:03.426 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_42 2023-10-11T11:08:03.427 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_replay_state cluster1 mirror image_42 started 2023-10-11T11:08:03.427 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:08:03.427 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:08:03.427 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_42 2023-10-11T11:08:03.428 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local state=started 2023-10-11T11:08:03.428 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T11:08:03.428 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-10-11T11:08:03.428 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 1 2023-10-11T11:08:04.427 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test_image_replay_state cluster1 mirror image_42 started 2023-10-11T11:08:04.427 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:08:04.427 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:08:04.428 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_42 2023-10-11T11:08:04.428 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local test_state=started 2023-10-11T11:08:04.428 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local status_result 2023-10-11T11:08:04.428 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_state=stopped 2023-10-11T11:08:04.432 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_42 2023-10-11T11:08:04.432 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ grep -i state 2023-10-11T11:08:04.432 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_instance=cluster1 2023-10-11T11:08:04.432 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T11:08:04.433 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:08:04.433 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance=cluster1 2023-10-11T11:08:04.433 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local loop_instance 2023-10-11T11:08:04.433 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-10-11T11:08:04.433 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sleep 0 2023-10-11T11:08:04.434 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-10-11T11:08:04.434 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ seq 0 3 2023-10-11T11:08:04.434 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-10-11T11:08:04.434 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_42 2023-10-11T11:08:04.434 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1:0 2023-10-11T11:08:04.434 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T11:08:04.435 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance 2023-10-11T11:08:04.435 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-10-11T11:08:04.435 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local val=cluster1:0 2023-10-11T11:08:04.435 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_var_name=cluster 2023-10-11T11:08:04.435 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance_var_name=instance 2023-10-11T11:08:04.436 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ cluster=cluster1 2023-10-11T11:08:04.436 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ instance=0 2023-10-11T11:08:04.436 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' 0 = cluster1:0 ']' 2023-10-11T11:08:04.436 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval cluster=cluster1 2023-10-11T11:08:04.436 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:08:04.437 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval instance=0 2023-10-11T11:08:04.437 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=0 2023-10-11T11:08:04.437 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-10-11T11:08:04.437 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local local_cluster=cluster1:0 2023-10-11T11:08:04.437 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster=cluster1 2023-10-11T11:08:04.437 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance 2023-10-11T11:08:04.438 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-10-11T11:08:04.438 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local val=cluster1:0 2023-10-11T11:08:04.438 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster_var_name=local_cluster 2023-10-11T11:08:04.438 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance_var_name=instance 2023-10-11T11:08:04.438 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:08:04.439 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=0 2023-10-11T11:08:04.439 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ '[' 0 = cluster1:0 ']' 2023-10-11T11:08:04.439 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval local_cluster=cluster1 2023-10-11T11:08:04.439 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ local_cluster=cluster1 2023-10-11T11:08:04.439 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval instance=0 2023-10-11T11:08:04.439 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ instance=0 2023-10-11T11:08:04.440 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-10-11T11:08:04.455 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T11:08:04.456 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T11:08:04.456 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T11:08:04.456 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_42 2023-10-11T11:08:04.556 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:Invalid command: unused arguments: ['mirror/image_42'] 2023-10-11T11:08:04.556 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd mirror status : get status for rbd mirror 2023-10-11T11:08:04.556 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:admin_socket: invalid command 2023-10-11T11:08:04.559 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-10-11T11:08:04.559 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_42 2023-10-11T11:08:04.560 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1:1 2023-10-11T11:08:04.560 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T11:08:04.560 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance 2023-10-11T11:08:04.560 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-10-11T11:08:04.560 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local val=cluster1:1 2023-10-11T11:08:04.561 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_var_name=cluster 2023-10-11T11:08:04.561 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance_var_name=instance 2023-10-11T11:08:04.561 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ cluster=cluster1 2023-10-11T11:08:04.561 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ instance=1 2023-10-11T11:08:04.561 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' 1 = cluster1:1 ']' 2023-10-11T11:08:04.561 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval cluster=cluster1 2023-10-11T11:08:04.562 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:08:04.562 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval instance=1 2023-10-11T11:08:04.562 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=1 2023-10-11T11:08:04.562 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-10-11T11:08:04.562 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local local_cluster=cluster1:1 2023-10-11T11:08:04.563 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster=cluster1 2023-10-11T11:08:04.563 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance 2023-10-11T11:08:04.563 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-10-11T11:08:04.563 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local val=cluster1:1 2023-10-11T11:08:04.563 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster_var_name=local_cluster 2023-10-11T11:08:04.564 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance_var_name=instance 2023-10-11T11:08:04.564 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:08:04.564 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=1 2023-10-11T11:08:04.564 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ '[' 1 = cluster1:1 ']' 2023-10-11T11:08:04.564 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval local_cluster=cluster1 2023-10-11T11:08:04.565 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ local_cluster=cluster1 2023-10-11T11:08:04.565 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval instance=1 2023-10-11T11:08:04.565 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ instance=1 2023-10-11T11:08:04.565 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-10-11T11:08:04.579 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T11:08:04.579 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T11:08:04.580 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T11:08:04.580 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_42 2023-10-11T11:08:04.729 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ return 0 2023-10-11T11:08:04.730 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ status_result=' "state": "Replaying"' 2023-10-11T11:08:04.730 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ echo ' "state": "Replaying"' 2023-10-11T11:08:04.731 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ grep -i Replaying 2023-10-11T11:08:04.731 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: "state": "Replaying" 2023-10-11T11:08:04.733 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ current_state=started 2023-10-11T11:08:04.733 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test started = started 2023-10-11T11:08:04.733 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:08:04.734 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_42 2023-10-11T11:08:04.734 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local local_cluster=cluster1 2023-10-11T11:08:04.734 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:08:04.734 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:08:04.734 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_42 2023-10-11T11:08:04.735 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' snapshot = journal ']' 2023-10-11T11:08:04.735 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' snapshot = snapshot ']' 2023-10-11T11:08:04.735 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_42 2023-10-11T11:08:04.735 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local local_cluster=cluster1 2023-10-11T11:08:04.735 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:08:04.736 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:08:04.736 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_42 2023-10-11T11:08:04.736 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ mkfname cluster2-mirror-image_42.status 2023-10-11T11:08:04.736 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ echo cluster2-mirror-image_42.status 2023-10-11T11:08:04.736 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -e 's|[/ ]|_|g' 2023-10-11T11:08:04.737 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local status_log=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_42.status 2023-10-11T11:08:04.737 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ mkfname cluster1-mirror-image_42.status 2023-10-11T11:08:04.738 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ echo cluster1-mirror-image_42.status 2023-10-11T11:08:04.738 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -e 's|[/ ]|_|g' 2023-10-11T11:08:04.738 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local local_status_log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_42.status 2023-10-11T11:08:04.738 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ mirror_image_snapshot cluster2 mirror image_42 2023-10-11T11:08:04.739 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:08:04.739 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:08:04.739 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_42 2023-10-11T11:08:04.739 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_42 2023-10-11T11:08:05.424 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:Snapshot ID: 201 2023-10-11T11:08:05.435 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_42 /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_42.status 2023-10-11T11:08:05.436 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:08:05.436 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:08:05.436 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_42 2023-10-11T11:08:05.436 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_42.status 2023-10-11T11:08:05.437 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_42 --format xml 2023-10-11T11:08:05.437 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:08:05.479 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-10-11T11:08:05.481 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snapshot_id=201 2023-10-11T11:08:05.481 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T11:08:05.481 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T11:08:05.482 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.2 2023-10-11T11:08:05.682 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_42 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_42.status 2023-10-11T11:08:05.683 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:08:05.683 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:08:05.683 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_42 2023-10-11T11:08:05.684 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_42.status 2023-10-11T11:08:05.684 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_42 --format xml 2023-10-11T11:08:05.684 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:08:05.729 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T11:08:05.731 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id=109 2023-10-11T11:08:05.731 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 201 = 109 2023-10-11T11:08:05.732 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T11:08:05.732 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.4 2023-10-11T11:08:06.132 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_42 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_42.status 2023-10-11T11:08:06.133 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:08:06.133 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:08:06.133 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_42 2023-10-11T11:08:06.133 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_42.status 2023-10-11T11:08:06.134 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_42 --format xml 2023-10-11T11:08:06.134 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:08:06.179 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T11:08:06.180 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T11:08:06.181 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:-:1.1: Document is empty 2023-10-11T11:08:06.181 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: 2023-10-11T11:08:06.181 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:^ 2023-10-11T11:08:06.182 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id= 2023-10-11T11:08:06.182 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 201 = '' 2023-10-11T11:08:06.182 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T11:08:06.183 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.8 2023-10-11T11:08:06.982 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_42 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_42.status 2023-10-11T11:08:06.983 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:08:06.983 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:08:06.983 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_42 2023-10-11T11:08:06.983 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_42.status 2023-10-11T11:08:06.983 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_42 --format xml 2023-10-11T11:08:06.984 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:08:07.055 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T11:08:07.056 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T11:08:07.057 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:-:1.1: Document is empty 2023-10-11T11:08:07.057 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: 2023-10-11T11:08:07.057 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:^ 2023-10-11T11:08:07.058 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id= 2023-10-11T11:08:07.058 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 201 = '' 2023-10-11T11:08:07.058 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T11:08:07.059 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 1.6 2023-10-11T11:08:08.659 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_42 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_42.status 2023-10-11T11:08:08.659 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:08:08.660 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:08:08.660 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_42 2023-10-11T11:08:08.660 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_42.status 2023-10-11T11:08:08.660 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_42 --format xml 2023-10-11T11:08:08.661 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:08:08.714 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T11:08:08.716 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id=201 2023-10-11T11:08:08.717 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 201 = 201 2023-10-11T11:08:08.717 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:08:08.717 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_snap_present cluster1 mirror image_42 snap 2023-10-11T11:08:08.717 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:08:08.718 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:08:08.718 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_42 2023-10-11T11:08:08.718 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap 2023-10-11T11:08:08.718 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T11:08:08.719 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-10-11T11:08:08.719 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 1 2023-10-11T11:08:09.717 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 info mirror/image_42@snap 2023-10-11T11:08:09.765 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:rbd image 'image_42': 2023-10-11T11:08:09.765 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: size 128 MiB in 32 objects 2023-10-11T11:08:09.765 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: order 22 (4 MiB objects) 2023-10-11T11:08:09.765 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: snapshot_count: 2 2023-10-11T11:08:09.766 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: id: 1289c4eed91d 2023-10-11T11:08:09.766 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: block_name_prefix: rbd_data.1289c4eed91d 2023-10-11T11:08:09.766 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: format: 2 2023-10-11T11:08:09.766 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-10-11T11:08:09.766 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: op_features: 2023-10-11T11:08:09.767 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: flags: 2023-10-11T11:08:09.767 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: create_timestamp: Wed Oct 11 11:00:47 2023 2023-10-11T11:08:09.767 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: access_timestamp: Wed Oct 11 11:00:47 2023 2023-10-11T11:08:09.767 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: modify_timestamp: Wed Oct 11 11:00:47 2023 2023-10-11T11:08:09.767 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: protected: False 2023-10-11T11:08:09.768 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring state: enabled 2023-10-11T11:08:09.768 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring mode: snapshot 2023-10-11T11:08:09.768 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring global id: 8fbaa844-5c7f-4d9f-b667-ac0ce0c88d90 2023-10-11T11:08:09.768 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring primary: false 2023-10-11T11:08:09.769 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:08:09.769 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ compare_image_snaps mirror image_42 snap 2023-10-11T11:08:09.770 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:08:09.770 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_42 2023-10-11T11:08:09.770 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap 2023-10-11T11:08:09.770 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local ret=0 2023-10-11T11:08:09.770 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local rmt_export=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_42.export 2023-10-11T11:08:09.771 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local loc_export=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_42.export 2023-10-11T11:08:09.771 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rm -f /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_42.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_42.export 2023-10-11T11:08:09.771 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 -p mirror export image_42@snap /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_42.export 2023-10-11T11:08:09.948 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-10-11T11:08:09.954 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 -p mirror export image_42@snap /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_42.export 2023-10-11T11:08:10.172 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-10-11T11:08:10.179 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ cmp /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_42.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_42.export 2023-10-11T11:08:10.234 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rm -f /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_42.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_42.export 2023-10-11T11:08:10.270 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:08:10.271 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T11:08:10.271 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_43 2023-10-11T11:08:10.271 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ create_snap cluster2 mirror image_43 snap 2023-10-11T11:08:10.271 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:08:10.271 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:08:10.272 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_43 2023-10-11T11:08:10.272 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap 2023-10-11T11:08:10.272 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 -p mirror snap create image_43@snap --debug-rbd=20 --debug-journaler=20 2023-10-11T11:08:11.442 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_replay_started cluster1 mirror image_43 2023-10-11T11:08:11.443 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:08:11.443 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:08:11.443 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_43 2023-10-11T11:08:11.443 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_replay_state cluster1 mirror image_43 started 2023-10-11T11:08:11.443 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:08:11.444 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:08:11.444 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_43 2023-10-11T11:08:11.444 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local state=started 2023-10-11T11:08:11.444 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T11:08:11.444 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-10-11T11:08:11.444 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 1 2023-10-11T11:08:12.444 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test_image_replay_state cluster1 mirror image_43 started 2023-10-11T11:08:12.444 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:08:12.444 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:08:12.444 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_43 2023-10-11T11:08:12.444 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local test_state=started 2023-10-11T11:08:12.445 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local status_result 2023-10-11T11:08:12.445 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_state=stopped 2023-10-11T11:08:12.446 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_43 2023-10-11T11:08:12.446 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_instance=cluster1 2023-10-11T11:08:12.446 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T11:08:12.446 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ grep -i state 2023-10-11T11:08:12.447 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:08:12.447 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance=cluster1 2023-10-11T11:08:12.447 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local loop_instance 2023-10-11T11:08:12.447 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-10-11T11:08:12.447 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sleep 0 2023-10-11T11:08:12.448 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-10-11T11:08:12.448 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ seq 0 3 2023-10-11T11:08:12.450 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-10-11T11:08:12.450 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_43 2023-10-11T11:08:12.450 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1:0 2023-10-11T11:08:12.450 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T11:08:12.451 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance 2023-10-11T11:08:12.451 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-10-11T11:08:12.451 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local val=cluster1:0 2023-10-11T11:08:12.451 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_var_name=cluster 2023-10-11T11:08:12.451 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance_var_name=instance 2023-10-11T11:08:12.451 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ cluster=cluster1 2023-10-11T11:08:12.452 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ instance=0 2023-10-11T11:08:12.452 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' 0 = cluster1:0 ']' 2023-10-11T11:08:12.452 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval cluster=cluster1 2023-10-11T11:08:12.452 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:08:12.452 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval instance=0 2023-10-11T11:08:12.452 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=0 2023-10-11T11:08:12.453 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-10-11T11:08:12.453 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local local_cluster=cluster1:0 2023-10-11T11:08:12.453 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster=cluster1 2023-10-11T11:08:12.453 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance 2023-10-11T11:08:12.453 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-10-11T11:08:12.453 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local val=cluster1:0 2023-10-11T11:08:12.454 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster_var_name=local_cluster 2023-10-11T11:08:12.455 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance_var_name=instance 2023-10-11T11:08:12.455 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:08:12.455 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=0 2023-10-11T11:08:12.455 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ '[' 0 = cluster1:0 ']' 2023-10-11T11:08:12.455 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval local_cluster=cluster1 2023-10-11T11:08:12.456 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ local_cluster=cluster1 2023-10-11T11:08:12.456 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval instance=0 2023-10-11T11:08:12.456 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ instance=0 2023-10-11T11:08:12.456 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-10-11T11:08:12.470 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T11:08:12.471 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T11:08:12.471 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T11:08:12.472 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_43 2023-10-11T11:08:12.572 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:Invalid command: unused arguments: ['mirror/image_43'] 2023-10-11T11:08:12.572 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd mirror status : get status for rbd mirror 2023-10-11T11:08:12.572 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:admin_socket: invalid command 2023-10-11T11:08:12.575 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-10-11T11:08:12.575 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_43 2023-10-11T11:08:12.575 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1:1 2023-10-11T11:08:12.575 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T11:08:12.576 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance 2023-10-11T11:08:12.576 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-10-11T11:08:12.576 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local val=cluster1:1 2023-10-11T11:08:12.576 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_var_name=cluster 2023-10-11T11:08:12.576 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance_var_name=instance 2023-10-11T11:08:12.576 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ cluster=cluster1 2023-10-11T11:08:12.577 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ instance=1 2023-10-11T11:08:12.577 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' 1 = cluster1:1 ']' 2023-10-11T11:08:12.577 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval cluster=cluster1 2023-10-11T11:08:12.577 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:08:12.577 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval instance=1 2023-10-11T11:08:12.578 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=1 2023-10-11T11:08:12.578 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-10-11T11:08:12.578 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local local_cluster=cluster1:1 2023-10-11T11:08:12.578 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster=cluster1 2023-10-11T11:08:12.578 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance 2023-10-11T11:08:12.579 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-10-11T11:08:12.579 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local val=cluster1:1 2023-10-11T11:08:12.579 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster_var_name=local_cluster 2023-10-11T11:08:12.579 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance_var_name=instance 2023-10-11T11:08:12.579 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:08:12.579 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=1 2023-10-11T11:08:12.580 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ '[' 1 = cluster1:1 ']' 2023-10-11T11:08:12.580 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval local_cluster=cluster1 2023-10-11T11:08:12.580 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ local_cluster=cluster1 2023-10-11T11:08:12.580 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval instance=1 2023-10-11T11:08:12.580 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ instance=1 2023-10-11T11:08:12.580 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-10-11T11:08:12.594 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T11:08:12.594 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T11:08:12.594 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T11:08:12.594 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_43 2023-10-11T11:08:12.743 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ return 0 2023-10-11T11:08:12.744 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ status_result=' "state": "Replaying"' 2023-10-11T11:08:12.744 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ echo ' "state": "Replaying"' 2023-10-11T11:08:12.745 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ grep -i Replaying 2023-10-11T11:08:12.745 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: "state": "Replaying" 2023-10-11T11:08:12.747 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ current_state=started 2023-10-11T11:08:12.747 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test started = started 2023-10-11T11:08:12.747 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:08:12.747 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_43 2023-10-11T11:08:12.747 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local local_cluster=cluster1 2023-10-11T11:08:12.748 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:08:12.748 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:08:12.748 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_43 2023-10-11T11:08:12.748 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' snapshot = journal ']' 2023-10-11T11:08:12.748 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' snapshot = snapshot ']' 2023-10-11T11:08:12.748 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_43 2023-10-11T11:08:12.749 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local local_cluster=cluster1 2023-10-11T11:08:12.749 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:08:12.749 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:08:12.749 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_43 2023-10-11T11:08:12.749 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ mkfname cluster2-mirror-image_43.status 2023-10-11T11:08:12.749 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ echo cluster2-mirror-image_43.status 2023-10-11T11:08:12.750 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -e 's|[/ ]|_|g' 2023-10-11T11:08:12.751 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local status_log=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_43.status 2023-10-11T11:08:12.751 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ mkfname cluster1-mirror-image_43.status 2023-10-11T11:08:12.751 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ echo cluster1-mirror-image_43.status 2023-10-11T11:08:12.751 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -e 's|[/ ]|_|g' 2023-10-11T11:08:12.752 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local local_status_log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_43.status 2023-10-11T11:08:12.752 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ mirror_image_snapshot cluster2 mirror image_43 2023-10-11T11:08:12.752 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:08:12.752 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:08:12.752 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_43 2023-10-11T11:08:12.752 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_43 2023-10-11T11:08:13.448 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:Snapshot ID: 203 2023-10-11T11:08:13.459 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_43 /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_43.status 2023-10-11T11:08:13.459 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:08:13.459 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:08:13.459 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_43 2023-10-11T11:08:13.459 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_43.status 2023-10-11T11:08:13.460 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_43 --format xml 2023-10-11T11:08:13.460 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:08:13.501 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-10-11T11:08:13.503 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snapshot_id=203 2023-10-11T11:08:13.503 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T11:08:13.504 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T11:08:13.504 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.2 2023-10-11T11:08:13.705 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_43 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_43.status 2023-10-11T11:08:13.705 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:08:13.705 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:08:13.705 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_43 2023-10-11T11:08:13.706 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_43.status 2023-10-11T11:08:13.706 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_43 --format xml 2023-10-11T11:08:13.706 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:08:13.751 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T11:08:13.753 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id=110 2023-10-11T11:08:13.753 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 203 = 110 2023-10-11T11:08:13.753 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T11:08:13.754 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.4 2023-10-11T11:08:14.155 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_43 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_43.status 2023-10-11T11:08:14.155 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:08:14.155 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:08:14.156 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_43 2023-10-11T11:08:14.156 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_43.status 2023-10-11T11:08:14.156 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_43 --format xml 2023-10-11T11:08:14.156 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:08:14.201 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T11:08:14.201 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T11:08:14.203 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:-:1.1: Document is empty 2023-10-11T11:08:14.203 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: 2023-10-11T11:08:14.203 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:^ 2023-10-11T11:08:14.203 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id= 2023-10-11T11:08:14.204 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 203 = '' 2023-10-11T11:08:14.204 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T11:08:14.204 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.8 2023-10-11T11:08:15.005 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_43 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_43.status 2023-10-11T11:08:15.005 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:08:15.005 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:08:15.006 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_43 2023-10-11T11:08:15.006 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_43.status 2023-10-11T11:08:15.006 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_43 --format xml 2023-10-11T11:08:15.007 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:08:15.080 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T11:08:15.080 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T11:08:15.082 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:-:1.1: Document is empty 2023-10-11T11:08:15.082 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: 2023-10-11T11:08:15.082 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:^ 2023-10-11T11:08:15.083 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id= 2023-10-11T11:08:15.084 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 203 = '' 2023-10-11T11:08:15.084 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T11:08:15.084 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 1.6 2023-10-11T11:08:16.684 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_43 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_43.status 2023-10-11T11:08:16.684 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:08:16.685 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:08:16.685 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_43 2023-10-11T11:08:16.685 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_43.status 2023-10-11T11:08:16.685 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_43 --format xml 2023-10-11T11:08:16.685 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:08:16.736 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T11:08:16.739 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id=203 2023-10-11T11:08:16.739 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 203 = 203 2023-10-11T11:08:16.739 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:08:16.739 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_snap_present cluster1 mirror image_43 snap 2023-10-11T11:08:16.739 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:08:16.740 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:08:16.740 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_43 2023-10-11T11:08:16.740 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap 2023-10-11T11:08:16.740 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T11:08:16.741 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-10-11T11:08:16.741 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 1 2023-10-11T11:08:17.739 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 info mirror/image_43@snap 2023-10-11T11:08:17.786 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:rbd image 'image_43': 2023-10-11T11:08:17.786 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: size 128 MiB in 32 objects 2023-10-11T11:08:17.787 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: order 22 (4 MiB objects) 2023-10-11T11:08:17.787 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: snapshot_count: 2 2023-10-11T11:08:17.787 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: id: 1289e756c5bf 2023-10-11T11:08:17.787 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: block_name_prefix: rbd_data.1289e756c5bf 2023-10-11T11:08:17.787 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: format: 2 2023-10-11T11:08:17.787 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-10-11T11:08:17.788 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: op_features: 2023-10-11T11:08:17.788 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: flags: 2023-10-11T11:08:17.788 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: create_timestamp: Wed Oct 11 11:00:55 2023 2023-10-11T11:08:17.788 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: access_timestamp: Wed Oct 11 11:00:55 2023 2023-10-11T11:08:17.788 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: modify_timestamp: Wed Oct 11 11:00:55 2023 2023-10-11T11:08:17.788 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: protected: False 2023-10-11T11:08:17.789 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring state: enabled 2023-10-11T11:08:17.789 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring mode: snapshot 2023-10-11T11:08:17.789 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring global id: 008ad2e7-4a4a-41fc-aeec-300676cc4f95 2023-10-11T11:08:17.789 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring primary: false 2023-10-11T11:08:17.790 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:08:17.790 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ compare_image_snaps mirror image_43 snap 2023-10-11T11:08:17.791 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:08:17.791 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_43 2023-10-11T11:08:17.791 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap 2023-10-11T11:08:17.791 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local ret=0 2023-10-11T11:08:17.791 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local rmt_export=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_43.export 2023-10-11T11:08:17.791 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local loc_export=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_43.export 2023-10-11T11:08:17.792 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rm -f /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_43.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_43.export 2023-10-11T11:08:17.792 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 -p mirror export image_43@snap /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_43.export 2023-10-11T11:08:17.964 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-10-11T11:08:17.969 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 -p mirror export image_43@snap /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_43.export 2023-10-11T11:08:18.395 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-10-11T11:08:18.401 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ cmp /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_43.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_43.export 2023-10-11T11:08:18.456 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rm -f /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_43.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_43.export 2023-10-11T11:08:18.493 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:08:18.493 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T11:08:18.493 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_44 2023-10-11T11:08:18.494 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ create_snap cluster2 mirror image_44 snap 2023-10-11T11:08:18.494 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:08:18.494 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:08:18.494 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_44 2023-10-11T11:08:18.495 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap 2023-10-11T11:08:18.495 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 -p mirror snap create image_44@snap --debug-rbd=20 --debug-journaler=20 2023-10-11T11:08:20.556 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_replay_started cluster1 mirror image_44 2023-10-11T11:08:20.556 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:08:20.557 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:08:20.557 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_44 2023-10-11T11:08:20.557 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_replay_state cluster1 mirror image_44 started 2023-10-11T11:08:20.557 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:08:20.558 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:08:20.558 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_44 2023-10-11T11:08:20.558 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local state=started 2023-10-11T11:08:20.559 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T11:08:20.559 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-10-11T11:08:20.559 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 1 2023-10-11T11:08:21.557 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test_image_replay_state cluster1 mirror image_44 started 2023-10-11T11:08:21.557 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:08:21.557 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:08:21.557 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_44 2023-10-11T11:08:21.557 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local test_state=started 2023-10-11T11:08:21.558 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local status_result 2023-10-11T11:08:21.558 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_state=stopped 2023-10-11T11:08:21.559 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_44 2023-10-11T11:08:21.559 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ grep -i state 2023-10-11T11:08:21.559 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_instance=cluster1 2023-10-11T11:08:21.560 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T11:08:21.560 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:08:21.560 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance=cluster1 2023-10-11T11:08:21.560 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local loop_instance 2023-10-11T11:08:21.560 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-10-11T11:08:21.561 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sleep 0 2023-10-11T11:08:21.561 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-10-11T11:08:21.561 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ seq 0 3 2023-10-11T11:08:21.562 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-10-11T11:08:21.563 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_44 2023-10-11T11:08:21.563 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1:0 2023-10-11T11:08:21.563 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T11:08:21.563 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance 2023-10-11T11:08:21.563 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-10-11T11:08:21.564 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local val=cluster1:0 2023-10-11T11:08:21.564 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_var_name=cluster 2023-10-11T11:08:21.564 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance_var_name=instance 2023-10-11T11:08:21.564 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ cluster=cluster1 2023-10-11T11:08:21.564 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ instance=0 2023-10-11T11:08:21.565 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' 0 = cluster1:0 ']' 2023-10-11T11:08:21.565 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval cluster=cluster1 2023-10-11T11:08:21.565 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:08:21.567 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval instance=0 2023-10-11T11:08:21.567 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=0 2023-10-11T11:08:21.567 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-10-11T11:08:21.567 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local local_cluster=cluster1:0 2023-10-11T11:08:21.567 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster=cluster1 2023-10-11T11:08:21.568 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance 2023-10-11T11:08:21.568 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-10-11T11:08:21.568 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local val=cluster1:0 2023-10-11T11:08:21.568 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster_var_name=local_cluster 2023-10-11T11:08:21.568 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance_var_name=instance 2023-10-11T11:08:21.569 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:08:21.569 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=0 2023-10-11T11:08:21.569 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ '[' 0 = cluster1:0 ']' 2023-10-11T11:08:21.569 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval local_cluster=cluster1 2023-10-11T11:08:21.569 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ local_cluster=cluster1 2023-10-11T11:08:21.569 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval instance=0 2023-10-11T11:08:21.570 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ instance=0 2023-10-11T11:08:21.570 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-10-11T11:08:21.585 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T11:08:21.586 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T11:08:21.586 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T11:08:21.586 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_44 2023-10-11T11:08:21.686 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:Invalid command: unused arguments: ['mirror/image_44'] 2023-10-11T11:08:21.686 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd mirror status : get status for rbd mirror 2023-10-11T11:08:21.687 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:admin_socket: invalid command 2023-10-11T11:08:21.689 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-10-11T11:08:21.690 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_44 2023-10-11T11:08:21.690 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1:1 2023-10-11T11:08:21.690 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T11:08:21.690 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance 2023-10-11T11:08:21.691 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-10-11T11:08:21.691 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local val=cluster1:1 2023-10-11T11:08:21.691 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_var_name=cluster 2023-10-11T11:08:21.691 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance_var_name=instance 2023-10-11T11:08:21.691 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ cluster=cluster1 2023-10-11T11:08:21.691 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ instance=1 2023-10-11T11:08:21.692 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' 1 = cluster1:1 ']' 2023-10-11T11:08:21.692 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval cluster=cluster1 2023-10-11T11:08:21.692 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:08:21.692 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval instance=1 2023-10-11T11:08:21.692 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=1 2023-10-11T11:08:21.693 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-10-11T11:08:21.693 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local local_cluster=cluster1:1 2023-10-11T11:08:21.693 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster=cluster1 2023-10-11T11:08:21.693 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance 2023-10-11T11:08:21.693 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-10-11T11:08:21.693 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local val=cluster1:1 2023-10-11T11:08:21.694 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster_var_name=local_cluster 2023-10-11T11:08:21.694 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance_var_name=instance 2023-10-11T11:08:21.694 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:08:21.694 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=1 2023-10-11T11:08:21.694 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ '[' 1 = cluster1:1 ']' 2023-10-11T11:08:21.695 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval local_cluster=cluster1 2023-10-11T11:08:21.695 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ local_cluster=cluster1 2023-10-11T11:08:21.695 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval instance=1 2023-10-11T11:08:21.695 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ instance=1 2023-10-11T11:08:21.695 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-10-11T11:08:21.708 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T11:08:21.709 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T11:08:21.709 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T11:08:21.709 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_44 2023-10-11T11:08:21.861 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ return 0 2023-10-11T11:08:21.862 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ status_result=' "state": "Replaying"' 2023-10-11T11:08:21.862 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ echo ' "state": "Replaying"' 2023-10-11T11:08:21.862 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ grep -i Replaying 2023-10-11T11:08:21.863 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: "state": "Replaying" 2023-10-11T11:08:21.864 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ current_state=started 2023-10-11T11:08:21.864 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test started = started 2023-10-11T11:08:21.864 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:08:21.865 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_44 2023-10-11T11:08:21.865 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local local_cluster=cluster1 2023-10-11T11:08:21.865 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:08:21.865 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:08:21.865 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_44 2023-10-11T11:08:21.866 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' snapshot = journal ']' 2023-10-11T11:08:21.866 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' snapshot = snapshot ']' 2023-10-11T11:08:21.866 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_44 2023-10-11T11:08:21.866 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local local_cluster=cluster1 2023-10-11T11:08:21.866 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:08:21.867 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:08:21.867 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_44 2023-10-11T11:08:21.867 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ mkfname cluster2-mirror-image_44.status 2023-10-11T11:08:21.867 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ echo cluster2-mirror-image_44.status 2023-10-11T11:08:21.867 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -e 's|[/ ]|_|g' 2023-10-11T11:08:21.868 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local status_log=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_44.status 2023-10-11T11:08:21.869 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ mkfname cluster1-mirror-image_44.status 2023-10-11T11:08:21.869 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ echo cluster1-mirror-image_44.status 2023-10-11T11:08:21.869 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -e 's|[/ ]|_|g' 2023-10-11T11:08:21.869 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local local_status_log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_44.status 2023-10-11T11:08:21.870 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ mirror_image_snapshot cluster2 mirror image_44 2023-10-11T11:08:21.870 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:08:21.870 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:08:21.870 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_44 2023-10-11T11:08:21.870 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_44 2023-10-11T11:08:22.563 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:Snapshot ID: 205 2023-10-11T11:08:22.576 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_44 /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_44.status 2023-10-11T11:08:22.576 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:08:22.576 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:08:22.576 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_44 2023-10-11T11:08:22.577 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_44.status 2023-10-11T11:08:22.577 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_44 --format xml 2023-10-11T11:08:22.577 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:08:22.619 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-10-11T11:08:22.621 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snapshot_id=205 2023-10-11T11:08:22.622 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T11:08:22.622 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T11:08:22.622 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.2 2023-10-11T11:08:22.823 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_44 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_44.status 2023-10-11T11:08:22.823 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:08:22.823 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:08:22.823 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_44 2023-10-11T11:08:22.823 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_44.status 2023-10-11T11:08:22.823 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_44 --format xml 2023-10-11T11:08:22.824 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:08:22.868 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T11:08:22.868 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T11:08:22.870 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:-:1.1: Document is empty 2023-10-11T11:08:22.870 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: 2023-10-11T11:08:22.870 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:^ 2023-10-11T11:08:22.870 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id= 2023-10-11T11:08:22.871 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 205 = '' 2023-10-11T11:08:22.871 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T11:08:22.871 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.4 2023-10-11T11:08:23.271 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_44 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_44.status 2023-10-11T11:08:23.271 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:08:23.272 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:08:23.272 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_44 2023-10-11T11:08:23.272 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_44.status 2023-10-11T11:08:23.272 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_44 --format xml 2023-10-11T11:08:23.272 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:08:23.316 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T11:08:23.317 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T11:08:23.318 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:-:1.1: Document is empty 2023-10-11T11:08:23.318 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: 2023-10-11T11:08:23.318 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:^ 2023-10-11T11:08:23.319 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id= 2023-10-11T11:08:23.319 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 205 = '' 2023-10-11T11:08:23.319 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T11:08:23.319 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.8 2023-10-11T11:08:24.120 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_44 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_44.status 2023-10-11T11:08:24.120 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:08:24.121 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:08:24.121 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_44 2023-10-11T11:08:24.121 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_44.status 2023-10-11T11:08:24.121 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_44 --format xml 2023-10-11T11:08:24.122 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:08:24.164 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T11:08:24.167 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id=205 2023-10-11T11:08:24.167 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 205 = 205 2023-10-11T11:08:24.167 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:08:24.167 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_snap_present cluster1 mirror image_44 snap 2023-10-11T11:08:24.168 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:08:24.168 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:08:24.168 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_44 2023-10-11T11:08:24.169 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap 2023-10-11T11:08:24.169 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T11:08:24.169 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-10-11T11:08:24.169 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 1 2023-10-11T11:08:25.167 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 info mirror/image_44@snap 2023-10-11T11:08:25.210 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:rbd image 'image_44': 2023-10-11T11:08:25.210 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: size 128 MiB in 32 objects 2023-10-11T11:08:25.211 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: order 22 (4 MiB objects) 2023-10-11T11:08:25.211 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: snapshot_count: 2 2023-10-11T11:08:25.211 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: id: 1289b960cb5d 2023-10-11T11:08:25.211 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: block_name_prefix: rbd_data.1289b960cb5d 2023-10-11T11:08:25.211 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: format: 2 2023-10-11T11:08:25.211 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-10-11T11:08:25.212 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: op_features: 2023-10-11T11:08:25.212 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: flags: 2023-10-11T11:08:25.212 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: create_timestamp: Wed Oct 11 11:01:10 2023 2023-10-11T11:08:25.212 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: access_timestamp: Wed Oct 11 11:01:10 2023 2023-10-11T11:08:25.212 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: modify_timestamp: Wed Oct 11 11:01:10 2023 2023-10-11T11:08:25.212 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: protected: False 2023-10-11T11:08:25.213 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring state: enabled 2023-10-11T11:08:25.213 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring mode: snapshot 2023-10-11T11:08:25.213 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring global id: 749fd0b3-e9a4-4bac-bd26-1b56437c5b8f 2023-10-11T11:08:25.213 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring primary: false 2023-10-11T11:08:25.214 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:08:25.214 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ compare_image_snaps mirror image_44 snap 2023-10-11T11:08:25.215 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:08:25.215 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_44 2023-10-11T11:08:25.215 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap 2023-10-11T11:08:25.215 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local ret=0 2023-10-11T11:08:25.215 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local rmt_export=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_44.export 2023-10-11T11:08:25.216 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local loc_export=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_44.export 2023-10-11T11:08:25.216 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rm -f /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_44.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_44.export 2023-10-11T11:08:25.216 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 -p mirror export image_44@snap /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_44.export 2023-10-11T11:08:25.401 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-10-11T11:08:25.408 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 -p mirror export image_44@snap /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_44.export 2023-10-11T11:08:25.832 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-10-11T11:08:25.839 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ cmp /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_44.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_44.export 2023-10-11T11:08:25.894 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rm -f /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_44.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_44.export 2023-10-11T11:08:25.930 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:08:25.931 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T11:08:25.931 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_45 2023-10-11T11:08:25.931 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ create_snap cluster2 mirror image_45 snap 2023-10-11T11:08:25.931 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:08:25.932 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:08:25.932 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_45 2023-10-11T11:08:25.932 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap 2023-10-11T11:08:25.932 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 -p mirror snap create image_45@snap --debug-rbd=20 --debug-journaler=20 2023-10-11T11:08:27.573 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_replay_started cluster1 mirror image_45 2023-10-11T11:08:27.573 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:08:27.574 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:08:27.574 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_45 2023-10-11T11:08:27.574 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_replay_state cluster1 mirror image_45 started 2023-10-11T11:08:27.574 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:08:27.574 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:08:27.574 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_45 2023-10-11T11:08:27.575 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local state=started 2023-10-11T11:08:27.575 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T11:08:27.575 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-10-11T11:08:27.575 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 1 2023-10-11T11:08:27.641 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2023-10-11T11:08:27.641 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2023-10-11T11:08:27.642 INFO:tasks.rbd_mirror_thrash:waiting for 0 secs before reviving daemons 2023-10-11T11:08:27.642 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2023-10-11T11:08:27.642 INFO:teuthology.orchestra.run:waiting for 600 2023-10-11T11:08:27.643 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi111.stderr:2023-10-11T11:08:27.641+0000 7f68529ff700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 119268) UID: 1000 2023-10-11T11:08:28.574 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test_image_replay_state cluster1 mirror image_45 started 2023-10-11T11:08:28.575 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:08:28.575 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:08:28.575 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_45 2023-10-11T11:08:28.575 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local test_state=started 2023-10-11T11:08:28.575 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local status_result 2023-10-11T11:08:28.576 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_state=stopped 2023-10-11T11:08:28.577 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_45 2023-10-11T11:08:28.577 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_instance=cluster1 2023-10-11T11:08:28.577 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T11:08:28.577 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:08:28.577 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance=cluster1 2023-10-11T11:08:28.578 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local loop_instance 2023-10-11T11:08:28.578 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-10-11T11:08:28.578 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sleep 0 2023-10-11T11:08:28.578 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ grep -i state 2023-10-11T11:08:28.578 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-10-11T11:08:28.579 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ seq 0 3 2023-10-11T11:08:28.580 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-10-11T11:08:28.580 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_45 2023-10-11T11:08:28.580 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1:0 2023-10-11T11:08:28.581 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T11:08:28.581 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance 2023-10-11T11:08:28.581 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-10-11T11:08:28.581 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local val=cluster1:0 2023-10-11T11:08:28.581 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_var_name=cluster 2023-10-11T11:08:28.581 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance_var_name=instance 2023-10-11T11:08:28.582 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ cluster=cluster1 2023-10-11T11:08:28.582 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ instance=0 2023-10-11T11:08:28.582 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' 0 = cluster1:0 ']' 2023-10-11T11:08:28.582 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval cluster=cluster1 2023-10-11T11:08:28.582 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:08:28.583 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval instance=0 2023-10-11T11:08:28.583 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=0 2023-10-11T11:08:28.584 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-10-11T11:08:28.584 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local local_cluster=cluster1:0 2023-10-11T11:08:28.584 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster=cluster1 2023-10-11T11:08:28.585 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance 2023-10-11T11:08:28.585 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-10-11T11:08:28.585 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local val=cluster1:0 2023-10-11T11:08:28.585 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster_var_name=local_cluster 2023-10-11T11:08:28.585 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance_var_name=instance 2023-10-11T11:08:28.586 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:08:28.586 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=0 2023-10-11T11:08:28.586 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ '[' 0 = cluster1:0 ']' 2023-10-11T11:08:28.586 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval local_cluster=cluster1 2023-10-11T11:08:28.586 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ local_cluster=cluster1 2023-10-11T11:08:28.586 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval instance=0 2023-10-11T11:08:28.587 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ instance=0 2023-10-11T11:08:28.587 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-10-11T11:08:28.602 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T11:08:28.603 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T11:08:28.603 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T11:08:28.603 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_45 2023-10-11T11:08:28.693 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:admin_socket: exception getting command descriptions: [Errno 2] No such file or directory 2023-10-11T11:08:28.697 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-10-11T11:08:28.697 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_45 2023-10-11T11:08:28.697 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1:1 2023-10-11T11:08:28.697 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T11:08:28.698 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance 2023-10-11T11:08:28.698 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-10-11T11:08:28.698 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local val=cluster1:1 2023-10-11T11:08:28.698 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_var_name=cluster 2023-10-11T11:08:28.698 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance_var_name=instance 2023-10-11T11:08:28.699 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ cluster=cluster1 2023-10-11T11:08:28.699 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ instance=1 2023-10-11T11:08:28.699 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' 1 = cluster1:1 ']' 2023-10-11T11:08:28.699 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval cluster=cluster1 2023-10-11T11:08:28.699 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:08:28.699 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval instance=1 2023-10-11T11:08:28.700 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=1 2023-10-11T11:08:28.700 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-10-11T11:08:28.700 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local local_cluster=cluster1:1 2023-10-11T11:08:28.700 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster=cluster1 2023-10-11T11:08:28.700 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance 2023-10-11T11:08:28.701 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-10-11T11:08:28.701 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local val=cluster1:1 2023-10-11T11:08:28.701 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster_var_name=local_cluster 2023-10-11T11:08:28.701 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance_var_name=instance 2023-10-11T11:08:28.701 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:08:28.702 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=1 2023-10-11T11:08:28.702 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ '[' 1 = cluster1:1 ']' 2023-10-11T11:08:28.702 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval local_cluster=cluster1 2023-10-11T11:08:28.702 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ local_cluster=cluster1 2023-10-11T11:08:28.702 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval instance=1 2023-10-11T11:08:28.702 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ instance=1 2023-10-11T11:08:28.703 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-10-11T11:08:28.716 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T11:08:28.716 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T11:08:28.717 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T11:08:28.717 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_45 2023-10-11T11:08:28.866 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ return 0 2023-10-11T11:08:28.867 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ status_result=' "state": "Replaying"' 2023-10-11T11:08:28.867 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ echo ' "state": "Replaying"' 2023-10-11T11:08:28.867 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ grep -i Replaying 2023-10-11T11:08:28.868 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: "state": "Replaying" 2023-10-11T11:08:28.869 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ current_state=started 2023-10-11T11:08:28.869 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test started = started 2023-10-11T11:08:28.870 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:08:28.870 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_45 2023-10-11T11:08:28.870 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local local_cluster=cluster1 2023-10-11T11:08:28.870 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:08:28.870 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:08:28.871 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_45 2023-10-11T11:08:28.871 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' snapshot = journal ']' 2023-10-11T11:08:28.871 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' snapshot = snapshot ']' 2023-10-11T11:08:28.871 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_45 2023-10-11T11:08:28.871 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local local_cluster=cluster1 2023-10-11T11:08:28.871 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:08:28.872 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:08:28.872 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_45 2023-10-11T11:08:28.872 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ mkfname cluster2-mirror-image_45.status 2023-10-11T11:08:28.872 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ echo cluster2-mirror-image_45.status 2023-10-11T11:08:28.872 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -e 's|[/ ]|_|g' 2023-10-11T11:08:28.874 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local status_log=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_45.status 2023-10-11T11:08:28.874 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ mkfname cluster1-mirror-image_45.status 2023-10-11T11:08:28.874 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ echo cluster1-mirror-image_45.status 2023-10-11T11:08:28.874 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -e 's|[/ ]|_|g' 2023-10-11T11:08:28.874 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local local_status_log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_45.status 2023-10-11T11:08:28.875 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ mirror_image_snapshot cluster2 mirror image_45 2023-10-11T11:08:28.875 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:08:28.875 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:08:28.875 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_45 2023-10-11T11:08:28.875 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_45 2023-10-11T11:08:29.577 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:Snapshot ID: 207 2023-10-11T11:08:29.590 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_45 /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_45.status 2023-10-11T11:08:29.590 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:08:29.590 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:08:29.591 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_45 2023-10-11T11:08:29.591 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_45.status 2023-10-11T11:08:29.592 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_45 --format xml 2023-10-11T11:08:29.592 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:08:29.635 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-10-11T11:08:29.637 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snapshot_id=207 2023-10-11T11:08:29.638 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T11:08:29.638 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T11:08:29.638 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.2 2023-10-11T11:08:29.838 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_45 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_45.status 2023-10-11T11:08:29.839 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:08:29.839 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:08:29.839 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_45 2023-10-11T11:08:29.839 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_45.status 2023-10-11T11:08:29.840 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_45 --format xml 2023-10-11T11:08:29.840 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:08:29.885 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T11:08:29.887 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id=112 2023-10-11T11:08:29.887 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 207 = 112 2023-10-11T11:08:29.887 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T11:08:29.887 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.4 2023-10-11T11:08:30.288 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_45 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_45.status 2023-10-11T11:08:30.288 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:08:30.288 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:08:30.289 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_45 2023-10-11T11:08:30.289 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_45.status 2023-10-11T11:08:30.289 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_45 --format xml 2023-10-11T11:08:30.289 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:08:30.335 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T11:08:30.335 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T11:08:30.336 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:-:1.1: Document is empty 2023-10-11T11:08:30.336 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: 2023-10-11T11:08:30.337 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:^ 2023-10-11T11:08:30.337 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id= 2023-10-11T11:08:30.337 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 207 = '' 2023-10-11T11:08:30.338 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T11:08:30.338 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.8 2023-10-11T11:08:31.138 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_45 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_45.status 2023-10-11T11:08:31.138 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:08:31.138 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:08:31.139 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_45 2023-10-11T11:08:31.139 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_45.status 2023-10-11T11:08:31.139 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_45 --format xml 2023-10-11T11:08:31.139 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:08:31.193 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T11:08:31.193 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T11:08:31.195 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:-:1.1: Document is empty 2023-10-11T11:08:31.195 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: 2023-10-11T11:08:31.195 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:^ 2023-10-11T11:08:31.195 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id= 2023-10-11T11:08:31.196 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 207 = '' 2023-10-11T11:08:31.196 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T11:08:31.196 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 1.6 2023-10-11T11:08:32.796 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_45 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_45.status 2023-10-11T11:08:32.797 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:08:32.797 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:08:32.797 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_45 2023-10-11T11:08:32.797 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_45.status 2023-10-11T11:08:32.798 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_45 --format xml 2023-10-11T11:08:32.798 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:08:32.847 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T11:08:32.850 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id=207 2023-10-11T11:08:32.850 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 207 = 207 2023-10-11T11:08:32.850 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:08:32.851 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_snap_present cluster1 mirror image_45 snap 2023-10-11T11:08:32.851 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:08:32.851 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:08:32.851 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_45 2023-10-11T11:08:32.851 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap 2023-10-11T11:08:32.851 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T11:08:32.852 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-10-11T11:08:32.852 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 1 2023-10-11T11:08:33.744 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2023-10-11T11:08:33.745 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2023-10-11T11:08:33.745 DEBUG:teuthology.orchestra.run.smithi111:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2023-10-11T11:08:33.749 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2023-10-11T11:08:33.749 INFO:tasks.rbd_mirror_thrash:waiting for 102 secs before thrashing 2023-10-11T11:08:33.850 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 info mirror/image_45@snap 2023-10-11T11:08:33.895 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:rbd image 'image_45': 2023-10-11T11:08:33.895 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: size 128 MiB in 32 objects 2023-10-11T11:08:33.895 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: order 22 (4 MiB objects) 2023-10-11T11:08:33.896 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: snapshot_count: 2 2023-10-11T11:08:33.896 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: id: 1289486349d8 2023-10-11T11:08:33.896 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: block_name_prefix: rbd_data.1289486349d8 2023-10-11T11:08:33.896 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: format: 2 2023-10-11T11:08:33.897 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-10-11T11:08:33.897 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: op_features: 2023-10-11T11:08:33.897 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: flags: 2023-10-11T11:08:33.897 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: create_timestamp: Wed Oct 11 11:01:24 2023 2023-10-11T11:08:33.898 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: access_timestamp: Wed Oct 11 11:01:24 2023 2023-10-11T11:08:33.898 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: modify_timestamp: Wed Oct 11 11:01:24 2023 2023-10-11T11:08:33.898 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: protected: False 2023-10-11T11:08:33.898 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring state: enabled 2023-10-11T11:08:33.899 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring mode: snapshot 2023-10-11T11:08:33.899 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring global id: fb355421-cd73-48b6-8cc6-59fefdadb677 2023-10-11T11:08:33.899 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring primary: false 2023-10-11T11:08:33.901 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:08:33.901 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ compare_image_snaps mirror image_45 snap 2023-10-11T11:08:33.901 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:08:33.902 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_45 2023-10-11T11:08:33.902 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap 2023-10-11T11:08:33.902 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local ret=0 2023-10-11T11:08:33.902 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local rmt_export=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_45.export 2023-10-11T11:08:33.903 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local loc_export=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_45.export 2023-10-11T11:08:33.903 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rm -f /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_45.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_45.export 2023-10-11T11:08:33.903 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 -p mirror export image_45@snap /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_45.export 2023-10-11T11:08:34.083 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-10-11T11:08:34.090 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 -p mirror export image_45@snap /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_45.export 2023-10-11T11:08:34.291 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-10-11T11:08:34.297 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ cmp /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_45.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_45.export 2023-10-11T11:08:34.352 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rm -f /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_45.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_45.export 2023-10-11T11:08:34.389 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:08:34.389 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T11:08:34.389 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_46 2023-10-11T11:08:34.390 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ create_snap cluster2 mirror image_46 snap 2023-10-11T11:08:34.390 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:08:34.390 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:08:34.391 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_46 2023-10-11T11:08:34.391 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap 2023-10-11T11:08:34.391 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 -p mirror snap create image_46@snap --debug-rbd=20 --debug-journaler=20 2023-10-11T11:08:35.586 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_replay_started cluster1 mirror image_46 2023-10-11T11:08:35.586 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:08:35.587 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:08:35.587 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_46 2023-10-11T11:08:35.587 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_replay_state cluster1 mirror image_46 started 2023-10-11T11:08:35.587 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:08:35.587 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:08:35.588 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_46 2023-10-11T11:08:35.588 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local state=started 2023-10-11T11:08:35.588 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T11:08:35.588 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-10-11T11:08:35.588 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 1 2023-10-11T11:08:36.588 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test_image_replay_state cluster1 mirror image_46 started 2023-10-11T11:08:36.588 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:08:36.589 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:08:36.589 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_46 2023-10-11T11:08:36.589 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local test_state=started 2023-10-11T11:08:36.590 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local status_result 2023-10-11T11:08:36.590 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_state=stopped 2023-10-11T11:08:36.590 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_46 2023-10-11T11:08:36.590 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_instance=cluster1 2023-10-11T11:08:36.591 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ grep -i state 2023-10-11T11:08:36.591 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T11:08:36.591 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:08:36.592 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance=cluster1 2023-10-11T11:08:36.592 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local loop_instance 2023-10-11T11:08:36.592 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-10-11T11:08:36.592 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sleep 0 2023-10-11T11:08:36.596 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-10-11T11:08:36.596 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ seq 0 3 2023-10-11T11:08:36.597 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-10-11T11:08:36.597 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_46 2023-10-11T11:08:36.597 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1:0 2023-10-11T11:08:36.598 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T11:08:36.598 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance 2023-10-11T11:08:36.598 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-10-11T11:08:36.598 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local val=cluster1:0 2023-10-11T11:08:36.599 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_var_name=cluster 2023-10-11T11:08:36.599 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance_var_name=instance 2023-10-11T11:08:36.599 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ cluster=cluster1 2023-10-11T11:08:36.600 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ instance=0 2023-10-11T11:08:36.600 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' 0 = cluster1:0 ']' 2023-10-11T11:08:36.600 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval cluster=cluster1 2023-10-11T11:08:36.600 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:08:36.601 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval instance=0 2023-10-11T11:08:36.601 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=0 2023-10-11T11:08:36.601 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-10-11T11:08:36.602 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local local_cluster=cluster1:0 2023-10-11T11:08:36.602 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster=cluster1 2023-10-11T11:08:36.602 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance 2023-10-11T11:08:36.602 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-10-11T11:08:36.603 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local val=cluster1:0 2023-10-11T11:08:36.603 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster_var_name=local_cluster 2023-10-11T11:08:36.603 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance_var_name=instance 2023-10-11T11:08:36.603 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:08:36.604 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=0 2023-10-11T11:08:36.604 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ '[' 0 = cluster1:0 ']' 2023-10-11T11:08:36.604 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval local_cluster=cluster1 2023-10-11T11:08:36.604 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ local_cluster=cluster1 2023-10-11T11:08:36.605 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval instance=0 2023-10-11T11:08:36.605 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ instance=0 2023-10-11T11:08:36.605 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-10-11T11:08:36.614 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T11:08:36.615 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T11:08:36.615 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T11:08:36.615 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_46 2023-10-11T11:08:36.715 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:Invalid command: unused arguments: ['mirror/image_46'] 2023-10-11T11:08:36.715 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd mirror status : get status for rbd mirror 2023-10-11T11:08:36.716 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:admin_socket: invalid command 2023-10-11T11:08:36.719 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-10-11T11:08:36.720 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_46 2023-10-11T11:08:36.720 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1:1 2023-10-11T11:08:36.720 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T11:08:36.720 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance 2023-10-11T11:08:36.721 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-10-11T11:08:36.721 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local val=cluster1:1 2023-10-11T11:08:36.721 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_var_name=cluster 2023-10-11T11:08:36.722 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance_var_name=instance 2023-10-11T11:08:36.722 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ cluster=cluster1 2023-10-11T11:08:36.722 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ instance=1 2023-10-11T11:08:36.722 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' 1 = cluster1:1 ']' 2023-10-11T11:08:36.723 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval cluster=cluster1 2023-10-11T11:08:36.723 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:08:36.723 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval instance=1 2023-10-11T11:08:36.723 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=1 2023-10-11T11:08:36.724 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-10-11T11:08:36.724 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local local_cluster=cluster1:1 2023-10-11T11:08:36.724 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster=cluster1 2023-10-11T11:08:36.725 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance 2023-10-11T11:08:36.725 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-10-11T11:08:36.725 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local val=cluster1:1 2023-10-11T11:08:36.725 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster_var_name=local_cluster 2023-10-11T11:08:36.726 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance_var_name=instance 2023-10-11T11:08:36.726 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:08:36.726 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=1 2023-10-11T11:08:36.726 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ '[' 1 = cluster1:1 ']' 2023-10-11T11:08:36.727 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval local_cluster=cluster1 2023-10-11T11:08:36.727 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ local_cluster=cluster1 2023-10-11T11:08:36.727 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval instance=1 2023-10-11T11:08:36.728 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ instance=1 2023-10-11T11:08:36.728 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-10-11T11:08:36.738 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T11:08:36.738 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T11:08:36.738 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T11:08:36.738 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_46 2023-10-11T11:08:36.886 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ return 0 2023-10-11T11:08:36.886 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ status_result=' "state": "Replaying"' 2023-10-11T11:08:36.887 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ echo ' "state": "Replaying"' 2023-10-11T11:08:36.887 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ grep -i Replaying 2023-10-11T11:08:36.888 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: "state": "Replaying" 2023-10-11T11:08:36.891 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ current_state=started 2023-10-11T11:08:36.891 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test started = started 2023-10-11T11:08:36.891 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:08:36.892 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_46 2023-10-11T11:08:36.892 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local local_cluster=cluster1 2023-10-11T11:08:36.892 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:08:36.893 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:08:36.893 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_46 2023-10-11T11:08:36.893 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' snapshot = journal ']' 2023-10-11T11:08:36.893 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' snapshot = snapshot ']' 2023-10-11T11:08:36.894 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_46 2023-10-11T11:08:36.894 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local local_cluster=cluster1 2023-10-11T11:08:36.894 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:08:36.895 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:08:36.895 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_46 2023-10-11T11:08:36.895 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ mkfname cluster2-mirror-image_46.status 2023-10-11T11:08:36.895 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ echo cluster2-mirror-image_46.status 2023-10-11T11:08:36.896 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -e 's|[/ ]|_|g' 2023-10-11T11:08:36.896 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local status_log=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_46.status 2023-10-11T11:08:36.896 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ mkfname cluster1-mirror-image_46.status 2023-10-11T11:08:36.896 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ echo cluster1-mirror-image_46.status 2023-10-11T11:08:36.897 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -e 's|[/ ]|_|g' 2023-10-11T11:08:36.897 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local local_status_log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_46.status 2023-10-11T11:08:36.897 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ mirror_image_snapshot cluster2 mirror image_46 2023-10-11T11:08:36.897 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:08:36.898 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:08:36.898 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_46 2023-10-11T11:08:36.898 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_46 2023-10-11T11:08:37.597 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:Snapshot ID: 209 2023-10-11T11:08:37.608 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_46 /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_46.status 2023-10-11T11:08:37.608 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:08:37.609 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:08:37.609 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_46 2023-10-11T11:08:37.609 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_46.status 2023-10-11T11:08:37.609 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_46 --format xml 2023-10-11T11:08:37.610 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:08:37.651 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-10-11T11:08:37.654 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snapshot_id=209 2023-10-11T11:08:37.654 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T11:08:37.655 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T11:08:37.655 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.2 2023-10-11T11:08:37.855 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_46 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_46.status 2023-10-11T11:08:37.855 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:08:37.855 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:08:37.856 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_46 2023-10-11T11:08:37.856 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_46.status 2023-10-11T11:08:37.856 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_46 --format xml 2023-10-11T11:08:37.856 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:08:37.900 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T11:08:37.900 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T11:08:37.902 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:-:1.1: Document is empty 2023-10-11T11:08:37.902 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: 2023-10-11T11:08:37.902 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:^ 2023-10-11T11:08:37.903 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id= 2023-10-11T11:08:37.903 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 209 = '' 2023-10-11T11:08:37.903 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T11:08:37.903 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.4 2023-10-11T11:08:38.303 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_46 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_46.status 2023-10-11T11:08:38.304 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:08:38.304 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:08:38.304 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_46 2023-10-11T11:08:38.305 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_46.status 2023-10-11T11:08:38.305 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_46 --format xml 2023-10-11T11:08:38.305 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:08:38.353 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T11:08:38.354 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T11:08:38.355 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:-:1.1: Document is empty 2023-10-11T11:08:38.355 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: 2023-10-11T11:08:38.355 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:^ 2023-10-11T11:08:38.356 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id= 2023-10-11T11:08:38.356 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 209 = '' 2023-10-11T11:08:38.357 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T11:08:38.357 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.8 2023-10-11T11:08:38.824 INFO:tasks.ceph.cluster2.osd.0.smithi111.stderr:2023-10-11T11:08:38.820+0000 7f80be969700 -1 reset not still connected to 0x5555f37df2b0 2023-10-11T11:08:38.824 INFO:tasks.ceph.cluster2.osd.0.smithi111.stderr:2023-10-11T11:08:38.820+0000 7f80be969700 -1 reset not still connected to 0x5555f37df450 2023-10-11T11:08:38.824 INFO:tasks.ceph.cluster2.osd.0.smithi111.stderr:2023-10-11T11:08:38.820+0000 7f80be969700 -1 reset not still connected to 0x5555f37df520 2023-10-11T11:08:38.824 INFO:tasks.ceph.cluster2.osd.0.smithi111.stderr:2023-10-11T11:08:38.820+0000 7f80be969700 -1 reset not still connected to 0x5555f37df5f0 2023-10-11T11:08:38.825 INFO:tasks.ceph.cluster2.osd.0.smithi111.stderr:2023-10-11T11:08:38.820+0000 7f80be969700 -1 reset not still connected to 0x5555f37df790 2023-10-11T11:08:38.825 INFO:tasks.ceph.cluster2.osd.0.smithi111.stderr:2023-10-11T11:08:38.820+0000 7f80be969700 -1 reset not still connected to 0x5555f37df930 2023-10-11T11:08:38.825 INFO:tasks.ceph.cluster2.osd.0.smithi111.stderr:2023-10-11T11:08:38.820+0000 7f80be969700 -1 reset not still connected to 0x5555f3b16d00 2023-10-11T11:08:38.825 INFO:tasks.ceph.cluster2.osd.0.smithi111.stderr:2023-10-11T11:08:38.820+0000 7f80be969700 -1 reset not still connected to 0x5555f3b17a00 2023-10-11T11:08:38.826 INFO:tasks.ceph.cluster2.osd.0.smithi111.stderr:2023-10-11T11:08:38.820+0000 7f80be969700 -1 reset not still connected to 0x5555f3b17ba0 2023-10-11T11:08:38.826 INFO:tasks.ceph.cluster2.osd.0.smithi111.stderr:2023-10-11T11:08:38.820+0000 7f80be969700 -1 reset not still connected to 0x5555fac4bd40 2023-10-11T11:08:38.826 INFO:tasks.ceph.cluster2.osd.0.smithi111.stderr:2023-10-11T11:08:38.820+0000 7f80be969700 -1 reset not still connected to 0x5555fbcff520 2023-10-11T11:08:38.826 INFO:tasks.ceph.cluster2.osd.0.smithi111.stderr:2023-10-11T11:08:38.820+0000 7f80be969700 -1 reset not still connected to 0x5555fe46b520 2023-10-11T11:08:38.827 INFO:tasks.ceph.cluster2.osd.0.smithi111.stderr:2023-10-11T11:08:38.820+0000 7f80be969700 -1 reset not still connected to 0x5555ffd79e10 2023-10-11T11:08:38.827 INFO:tasks.ceph.cluster2.osd.0.smithi111.stderr:2023-10-11T11:08:38.820+0000 7f80be969700 -1 reset not still connected to 0x5556004ba8f0 2023-10-11T11:08:39.157 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_46 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_46.status 2023-10-11T11:08:39.157 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:08:39.158 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:08:39.158 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_46 2023-10-11T11:08:39.158 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_46.status 2023-10-11T11:08:39.159 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_46 --format xml 2023-10-11T11:08:39.159 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:08:39.202 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T11:08:39.204 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id=209 2023-10-11T11:08:39.204 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 209 = 209 2023-10-11T11:08:39.205 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:08:39.205 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_snap_present cluster1 mirror image_46 snap 2023-10-11T11:08:39.205 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:08:39.205 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:08:39.206 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_46 2023-10-11T11:08:39.206 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap 2023-10-11T11:08:39.206 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T11:08:39.206 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-10-11T11:08:39.207 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 1 2023-10-11T11:08:40.204 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 info mirror/image_46@snap 2023-10-11T11:08:40.250 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:rbd image 'image_46': 2023-10-11T11:08:40.250 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: size 128 MiB in 32 objects 2023-10-11T11:08:40.250 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: order 22 (4 MiB objects) 2023-10-11T11:08:40.250 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: snapshot_count: 2 2023-10-11T11:08:40.250 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: id: 1289a7701286 2023-10-11T11:08:40.251 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: block_name_prefix: rbd_data.1289a7701286 2023-10-11T11:08:40.251 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: format: 2 2023-10-11T11:08:40.251 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-10-11T11:08:40.251 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: op_features: 2023-10-11T11:08:40.251 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: flags: 2023-10-11T11:08:40.251 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: create_timestamp: Wed Oct 11 11:01:31 2023 2023-10-11T11:08:40.252 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: access_timestamp: Wed Oct 11 11:01:31 2023 2023-10-11T11:08:40.252 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: modify_timestamp: Wed Oct 11 11:01:31 2023 2023-10-11T11:08:40.252 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: protected: False 2023-10-11T11:08:40.252 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring state: enabled 2023-10-11T11:08:40.252 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring mode: snapshot 2023-10-11T11:08:40.252 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring global id: 96075109-54b3-44cf-8c55-4ffc63857fb0 2023-10-11T11:08:40.253 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring primary: false 2023-10-11T11:08:40.254 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:08:40.254 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ compare_image_snaps mirror image_46 snap 2023-10-11T11:08:40.254 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:08:40.254 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_46 2023-10-11T11:08:40.254 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap 2023-10-11T11:08:40.255 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local ret=0 2023-10-11T11:08:40.255 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local rmt_export=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_46.export 2023-10-11T11:08:40.255 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local loc_export=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_46.export 2023-10-11T11:08:40.255 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rm -f /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_46.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_46.export 2023-10-11T11:08:40.255 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 -p mirror export image_46@snap /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_46.export 2023-10-11T11:08:40.432 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-10-11T11:08:40.438 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 -p mirror export image_46@snap /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_46.export 2023-10-11T11:08:40.668 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-10-11T11:08:40.675 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ cmp /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_46.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_46.export 2023-10-11T11:08:40.732 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rm -f /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_46.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_46.export 2023-10-11T11:08:40.768 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:08:40.769 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T11:08:40.769 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_47 2023-10-11T11:08:40.769 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ create_snap cluster2 mirror image_47 snap 2023-10-11T11:08:40.769 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:08:40.770 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:08:40.770 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_47 2023-10-11T11:08:40.770 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap 2023-10-11T11:08:40.771 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 -p mirror snap create image_47@snap --debug-rbd=20 --debug-journaler=20 2023-10-11T11:08:42.449 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_replay_started cluster1 mirror image_47 2023-10-11T11:08:42.449 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:08:42.450 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:08:42.450 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_47 2023-10-11T11:08:42.450 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_replay_state cluster1 mirror image_47 started 2023-10-11T11:08:42.450 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:08:42.451 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:08:42.451 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_47 2023-10-11T11:08:42.451 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local state=started 2023-10-11T11:08:42.451 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T11:08:42.452 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-10-11T11:08:42.452 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 1 2023-10-11T11:08:43.451 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test_image_replay_state cluster1 mirror image_47 started 2023-10-11T11:08:43.452 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:08:43.452 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:08:43.452 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_47 2023-10-11T11:08:43.453 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local test_state=started 2023-10-11T11:08:43.453 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local status_result 2023-10-11T11:08:43.453 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_state=stopped 2023-10-11T11:08:43.453 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_47 2023-10-11T11:08:43.454 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_instance=cluster1 2023-10-11T11:08:43.454 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T11:08:43.454 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ grep -i state 2023-10-11T11:08:43.455 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:08:43.455 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance=cluster1 2023-10-11T11:08:43.455 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local loop_instance 2023-10-11T11:08:43.455 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-10-11T11:08:43.456 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sleep 0 2023-10-11T11:08:43.456 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-10-11T11:08:43.460 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ seq 0 3 2023-10-11T11:08:43.460 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-10-11T11:08:43.460 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_47 2023-10-11T11:08:43.461 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1:0 2023-10-11T11:08:43.461 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T11:08:43.461 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance 2023-10-11T11:08:43.462 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-10-11T11:08:43.462 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local val=cluster1:0 2023-10-11T11:08:43.462 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_var_name=cluster 2023-10-11T11:08:43.462 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance_var_name=instance 2023-10-11T11:08:43.463 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ cluster=cluster1 2023-10-11T11:08:43.463 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ instance=0 2023-10-11T11:08:43.463 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' 0 = cluster1:0 ']' 2023-10-11T11:08:43.463 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval cluster=cluster1 2023-10-11T11:08:43.464 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:08:43.464 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval instance=0 2023-10-11T11:08:43.464 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=0 2023-10-11T11:08:43.464 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-10-11T11:08:43.465 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local local_cluster=cluster1:0 2023-10-11T11:08:43.465 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster=cluster1 2023-10-11T11:08:43.465 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance 2023-10-11T11:08:43.465 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-10-11T11:08:43.466 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local val=cluster1:0 2023-10-11T11:08:43.466 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster_var_name=local_cluster 2023-10-11T11:08:43.466 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance_var_name=instance 2023-10-11T11:08:43.466 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:08:43.467 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=0 2023-10-11T11:08:43.467 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ '[' 0 = cluster1:0 ']' 2023-10-11T11:08:43.467 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval local_cluster=cluster1 2023-10-11T11:08:43.467 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ local_cluster=cluster1 2023-10-11T11:08:43.468 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval instance=0 2023-10-11T11:08:43.468 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ instance=0 2023-10-11T11:08:43.468 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-10-11T11:08:43.477 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T11:08:43.477 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T11:08:43.478 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T11:08:43.478 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_47 2023-10-11T11:08:43.578 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:Invalid command: unused arguments: ['mirror/image_47'] 2023-10-11T11:08:43.578 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd mirror status : get status for rbd mirror 2023-10-11T11:08:43.578 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:admin_socket: invalid command 2023-10-11T11:08:43.582 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-10-11T11:08:43.582 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_47 2023-10-11T11:08:43.583 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1:1 2023-10-11T11:08:43.583 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T11:08:43.583 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance 2023-10-11T11:08:43.584 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-10-11T11:08:43.584 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local val=cluster1:1 2023-10-11T11:08:43.584 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_var_name=cluster 2023-10-11T11:08:43.584 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance_var_name=instance 2023-10-11T11:08:43.585 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ cluster=cluster1 2023-10-11T11:08:43.585 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ instance=1 2023-10-11T11:08:43.585 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' 1 = cluster1:1 ']' 2023-10-11T11:08:43.585 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval cluster=cluster1 2023-10-11T11:08:43.586 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:08:43.586 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval instance=1 2023-10-11T11:08:43.586 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=1 2023-10-11T11:08:43.587 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-10-11T11:08:43.587 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local local_cluster=cluster1:1 2023-10-11T11:08:43.587 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster=cluster1 2023-10-11T11:08:43.587 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance 2023-10-11T11:08:43.588 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-10-11T11:08:43.588 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local val=cluster1:1 2023-10-11T11:08:43.588 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster_var_name=local_cluster 2023-10-11T11:08:43.588 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance_var_name=instance 2023-10-11T11:08:43.589 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:08:43.589 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=1 2023-10-11T11:08:43.589 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ '[' 1 = cluster1:1 ']' 2023-10-11T11:08:43.590 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval local_cluster=cluster1 2023-10-11T11:08:43.590 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ local_cluster=cluster1 2023-10-11T11:08:43.590 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval instance=1 2023-10-11T11:08:43.590 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ instance=1 2023-10-11T11:08:43.591 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-10-11T11:08:43.601 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T11:08:43.601 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T11:08:43.602 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T11:08:43.602 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_47 2023-10-11T11:08:43.753 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ return 0 2023-10-11T11:08:43.754 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ status_result=' "state": "Replaying"' 2023-10-11T11:08:43.754 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ echo ' "state": "Replaying"' 2023-10-11T11:08:43.755 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ grep -i Replaying 2023-10-11T11:08:43.755 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: "state": "Replaying" 2023-10-11T11:08:43.759 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ current_state=started 2023-10-11T11:08:43.759 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test started = started 2023-10-11T11:08:43.759 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:08:43.760 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_47 2023-10-11T11:08:43.760 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local local_cluster=cluster1 2023-10-11T11:08:43.760 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:08:43.760 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:08:43.761 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_47 2023-10-11T11:08:43.761 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' snapshot = journal ']' 2023-10-11T11:08:43.761 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' snapshot = snapshot ']' 2023-10-11T11:08:43.761 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_47 2023-10-11T11:08:43.762 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local local_cluster=cluster1 2023-10-11T11:08:43.762 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:08:43.762 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:08:43.763 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_47 2023-10-11T11:08:43.763 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ mkfname cluster2-mirror-image_47.status 2023-10-11T11:08:43.763 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ echo cluster2-mirror-image_47.status 2023-10-11T11:08:43.763 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -e 's|[/ ]|_|g' 2023-10-11T11:08:43.764 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local status_log=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_47.status 2023-10-11T11:08:43.764 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ mkfname cluster1-mirror-image_47.status 2023-10-11T11:08:43.764 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ echo cluster1-mirror-image_47.status 2023-10-11T11:08:43.764 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -e 's|[/ ]|_|g' 2023-10-11T11:08:43.765 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local local_status_log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_47.status 2023-10-11T11:08:43.765 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ mirror_image_snapshot cluster2 mirror image_47 2023-10-11T11:08:43.765 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:08:43.765 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:08:43.766 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_47 2023-10-11T11:08:43.766 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_47 2023-10-11T11:08:44.456 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:Snapshot ID: 211 2023-10-11T11:08:44.468 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_47 /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_47.status 2023-10-11T11:08:44.468 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:08:44.468 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:08:44.469 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_47 2023-10-11T11:08:44.469 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_47.status 2023-10-11T11:08:44.469 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_47 --format xml 2023-10-11T11:08:44.469 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:08:44.511 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-10-11T11:08:44.513 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snapshot_id=211 2023-10-11T11:08:44.513 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T11:08:44.514 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T11:08:44.514 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.2 2023-10-11T11:08:44.714 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_47 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_47.status 2023-10-11T11:08:44.715 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:08:44.715 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:08:44.716 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_47 2023-10-11T11:08:44.716 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_47.status 2023-10-11T11:08:44.716 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_47 --format xml 2023-10-11T11:08:44.716 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:08:44.764 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T11:08:44.764 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T11:08:44.765 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:-:1.1: Document is empty 2023-10-11T11:08:44.766 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: 2023-10-11T11:08:44.766 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:^ 2023-10-11T11:08:44.766 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id= 2023-10-11T11:08:44.767 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 211 = '' 2023-10-11T11:08:44.767 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T11:08:44.767 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.4 2023-10-11T11:08:45.167 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_47 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_47.status 2023-10-11T11:08:45.167 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:08:45.168 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:08:45.168 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_47 2023-10-11T11:08:45.168 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_47.status 2023-10-11T11:08:45.168 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_47 --format xml 2023-10-11T11:08:45.169 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:08:45.214 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T11:08:45.214 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T11:08:45.215 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:-:1.1: Document is empty 2023-10-11T11:08:45.216 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: 2023-10-11T11:08:45.216 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:^ 2023-10-11T11:08:45.217 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id= 2023-10-11T11:08:45.217 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 211 = '' 2023-10-11T11:08:45.217 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T11:08:45.218 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.8 2023-10-11T11:08:46.017 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_47 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_47.status 2023-10-11T11:08:46.018 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:08:46.018 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:08:46.018 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_47 2023-10-11T11:08:46.018 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_47.status 2023-10-11T11:08:46.019 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_47 --format xml 2023-10-11T11:08:46.019 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:08:46.063 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T11:08:46.066 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id=211 2023-10-11T11:08:46.066 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 211 = 211 2023-10-11T11:08:46.066 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:08:46.066 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_snap_present cluster1 mirror image_47 snap 2023-10-11T11:08:46.067 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:08:46.067 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:08:46.067 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_47 2023-10-11T11:08:46.067 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap 2023-10-11T11:08:46.068 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T11:08:46.068 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-10-11T11:08:46.068 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 1 2023-10-11T11:08:47.066 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 info mirror/image_47@snap 2023-10-11T11:08:47.125 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:rbd image 'image_47': 2023-10-11T11:08:47.125 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: size 128 MiB in 32 objects 2023-10-11T11:08:47.125 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: order 22 (4 MiB objects) 2023-10-11T11:08:47.125 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: snapshot_count: 2 2023-10-11T11:08:47.126 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: id: 1289d8a9b317 2023-10-11T11:08:47.126 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: block_name_prefix: rbd_data.1289d8a9b317 2023-10-11T11:08:47.126 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: format: 2 2023-10-11T11:08:47.126 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-10-11T11:08:47.126 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: op_features: 2023-10-11T11:08:47.127 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: flags: 2023-10-11T11:08:47.127 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: create_timestamp: Wed Oct 11 11:01:42 2023 2023-10-11T11:08:47.127 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: access_timestamp: Wed Oct 11 11:01:42 2023 2023-10-11T11:08:47.127 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: modify_timestamp: Wed Oct 11 11:01:42 2023 2023-10-11T11:08:47.127 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: protected: False 2023-10-11T11:08:47.127 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring state: enabled 2023-10-11T11:08:47.128 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring mode: snapshot 2023-10-11T11:08:47.128 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring global id: 76f1a297-bf03-44bd-a55f-4ceb8d92fdbd 2023-10-11T11:08:47.128 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring primary: false 2023-10-11T11:08:47.129 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:08:47.129 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ compare_image_snaps mirror image_47 snap 2023-10-11T11:08:47.129 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:08:47.129 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_47 2023-10-11T11:08:47.130 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap 2023-10-11T11:08:47.130 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local ret=0 2023-10-11T11:08:47.130 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local rmt_export=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_47.export 2023-10-11T11:08:47.130 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local loc_export=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_47.export 2023-10-11T11:08:47.130 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rm -f /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_47.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_47.export 2023-10-11T11:08:47.131 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 -p mirror export image_47@snap /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_47.export 2023-10-11T11:08:47.314 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-10-11T11:08:47.320 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 -p mirror export image_47@snap /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_47.export 2023-10-11T11:08:47.554 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-10-11T11:08:47.561 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ cmp /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_47.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_47.export 2023-10-11T11:08:47.615 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rm -f /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_47.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_47.export 2023-10-11T11:08:47.651 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:08:47.652 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T11:08:47.652 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_48 2023-10-11T11:08:47.652 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ create_snap cluster2 mirror image_48 snap 2023-10-11T11:08:47.653 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:08:47.653 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:08:47.653 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_48 2023-10-11T11:08:47.653 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap 2023-10-11T11:08:47.653 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 -p mirror snap create image_48@snap --debug-rbd=20 --debug-journaler=20 2023-10-11T11:08:49.554 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_replay_started cluster1 mirror image_48 2023-10-11T11:08:49.555 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:08:49.555 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:08:49.555 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_48 2023-10-11T11:08:49.555 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_replay_state cluster1 mirror image_48 started 2023-10-11T11:08:49.556 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:08:49.556 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:08:49.556 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_48 2023-10-11T11:08:49.556 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local state=started 2023-10-11T11:08:49.556 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T11:08:49.556 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-10-11T11:08:49.557 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 1 2023-10-11T11:08:50.555 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test_image_replay_state cluster1 mirror image_48 started 2023-10-11T11:08:50.556 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:08:50.556 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:08:50.556 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_48 2023-10-11T11:08:50.556 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local test_state=started 2023-10-11T11:08:50.557 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local status_result 2023-10-11T11:08:50.557 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_state=stopped 2023-10-11T11:08:50.558 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_48 2023-10-11T11:08:50.558 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_instance=cluster1 2023-10-11T11:08:50.558 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T11:08:50.558 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ grep -i state 2023-10-11T11:08:50.559 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:08:50.559 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance=cluster1 2023-10-11T11:08:50.559 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local loop_instance 2023-10-11T11:08:50.559 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-10-11T11:08:50.559 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sleep 0 2023-10-11T11:08:50.560 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-10-11T11:08:50.560 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ seq 0 3 2023-10-11T11:08:50.562 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-10-11T11:08:50.563 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_48 2023-10-11T11:08:50.563 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1:0 2023-10-11T11:08:50.563 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T11:08:50.563 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance 2023-10-11T11:08:50.563 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-10-11T11:08:50.564 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local val=cluster1:0 2023-10-11T11:08:50.564 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_var_name=cluster 2023-10-11T11:08:50.564 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance_var_name=instance 2023-10-11T11:08:50.564 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ cluster=cluster1 2023-10-11T11:08:50.564 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ instance=0 2023-10-11T11:08:50.564 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' 0 = cluster1:0 ']' 2023-10-11T11:08:50.565 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval cluster=cluster1 2023-10-11T11:08:50.565 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:08:50.565 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval instance=0 2023-10-11T11:08:50.565 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=0 2023-10-11T11:08:50.565 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-10-11T11:08:50.565 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local local_cluster=cluster1:0 2023-10-11T11:08:50.566 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster=cluster1 2023-10-11T11:08:50.566 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance 2023-10-11T11:08:50.566 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-10-11T11:08:50.566 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local val=cluster1:0 2023-10-11T11:08:50.566 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster_var_name=local_cluster 2023-10-11T11:08:50.567 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance_var_name=instance 2023-10-11T11:08:50.567 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:08:50.567 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=0 2023-10-11T11:08:50.567 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ '[' 0 = cluster1:0 ']' 2023-10-11T11:08:50.567 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval local_cluster=cluster1 2023-10-11T11:08:50.568 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ local_cluster=cluster1 2023-10-11T11:08:50.568 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval instance=0 2023-10-11T11:08:50.568 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ instance=0 2023-10-11T11:08:50.568 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-10-11T11:08:50.584 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T11:08:50.584 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T11:08:50.584 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T11:08:50.585 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_48 2023-10-11T11:08:50.685 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:Invalid command: unused arguments: ['mirror/image_48'] 2023-10-11T11:08:50.685 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd mirror status : get status for rbd mirror 2023-10-11T11:08:50.686 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:admin_socket: invalid command 2023-10-11T11:08:50.688 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-10-11T11:08:50.689 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_48 2023-10-11T11:08:50.689 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1:1 2023-10-11T11:08:50.689 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T11:08:50.689 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance 2023-10-11T11:08:50.689 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-10-11T11:08:50.689 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local val=cluster1:1 2023-10-11T11:08:50.690 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_var_name=cluster 2023-10-11T11:08:50.690 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance_var_name=instance 2023-10-11T11:08:50.690 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ cluster=cluster1 2023-10-11T11:08:50.690 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ instance=1 2023-10-11T11:08:50.690 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' 1 = cluster1:1 ']' 2023-10-11T11:08:50.691 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval cluster=cluster1 2023-10-11T11:08:50.691 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:08:50.691 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval instance=1 2023-10-11T11:08:50.691 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=1 2023-10-11T11:08:50.691 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-10-11T11:08:50.691 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local local_cluster=cluster1:1 2023-10-11T11:08:50.692 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster=cluster1 2023-10-11T11:08:50.692 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance 2023-10-11T11:08:50.692 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-10-11T11:08:50.692 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local val=cluster1:1 2023-10-11T11:08:50.692 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster_var_name=local_cluster 2023-10-11T11:08:50.692 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance_var_name=instance 2023-10-11T11:08:50.693 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:08:50.693 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=1 2023-10-11T11:08:50.693 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ '[' 1 = cluster1:1 ']' 2023-10-11T11:08:50.693 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval local_cluster=cluster1 2023-10-11T11:08:50.693 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ local_cluster=cluster1 2023-10-11T11:08:50.694 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval instance=1 2023-10-11T11:08:50.694 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ instance=1 2023-10-11T11:08:50.694 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-10-11T11:08:50.708 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T11:08:50.708 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T11:08:50.709 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T11:08:50.709 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_48 2023-10-11T11:08:50.861 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ return 0 2023-10-11T11:08:50.862 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ status_result=' "state": "Replaying"' 2023-10-11T11:08:50.862 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ echo ' "state": "Replaying"' 2023-10-11T11:08:50.862 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ grep -i Replaying 2023-10-11T11:08:50.863 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: "state": "Replaying" 2023-10-11T11:08:50.865 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ current_state=started 2023-10-11T11:08:50.865 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test started = started 2023-10-11T11:08:50.865 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:08:50.865 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_48 2023-10-11T11:08:50.865 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local local_cluster=cluster1 2023-10-11T11:08:50.866 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:08:50.866 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:08:50.866 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_48 2023-10-11T11:08:50.866 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' snapshot = journal ']' 2023-10-11T11:08:50.866 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' snapshot = snapshot ']' 2023-10-11T11:08:50.867 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_48 2023-10-11T11:08:50.867 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local local_cluster=cluster1 2023-10-11T11:08:50.867 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:08:50.867 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:08:50.867 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_48 2023-10-11T11:08:50.868 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ mkfname cluster2-mirror-image_48.status 2023-10-11T11:08:50.868 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ echo cluster2-mirror-image_48.status 2023-10-11T11:08:50.868 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -e 's|[/ ]|_|g' 2023-10-11T11:08:50.869 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local status_log=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_48.status 2023-10-11T11:08:50.869 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ mkfname cluster1-mirror-image_48.status 2023-10-11T11:08:50.869 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ echo cluster1-mirror-image_48.status 2023-10-11T11:08:50.870 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -e 's|[/ ]|_|g' 2023-10-11T11:08:50.870 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local local_status_log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_48.status 2023-10-11T11:08:50.870 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ mirror_image_snapshot cluster2 mirror image_48 2023-10-11T11:08:50.870 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:08:50.870 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:08:50.871 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_48 2023-10-11T11:08:50.871 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_48 2023-10-11T11:08:51.550 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:Snapshot ID: 213 2023-10-11T11:08:51.560 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_48 /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_48.status 2023-10-11T11:08:51.561 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:08:51.561 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:08:51.561 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_48 2023-10-11T11:08:51.561 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_48.status 2023-10-11T11:08:51.561 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_48 --format xml 2023-10-11T11:08:51.562 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:08:51.604 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-10-11T11:08:51.606 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snapshot_id=213 2023-10-11T11:08:51.606 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T11:08:51.606 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T11:08:51.606 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.2 2023-10-11T11:08:51.807 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_48 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_48.status 2023-10-11T11:08:51.807 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:08:51.807 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:08:51.808 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_48 2023-10-11T11:08:51.808 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_48.status 2023-10-11T11:08:51.808 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_48 --format xml 2023-10-11T11:08:51.808 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:08:51.851 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T11:08:51.852 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T11:08:51.853 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:-:1.1: Document is empty 2023-10-11T11:08:51.853 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: 2023-10-11T11:08:51.853 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:^ 2023-10-11T11:08:51.854 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id= 2023-10-11T11:08:51.854 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 213 = '' 2023-10-11T11:08:51.855 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T11:08:51.855 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.4 2023-10-11T11:08:52.255 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_48 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_48.status 2023-10-11T11:08:52.255 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:08:52.255 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:08:52.255 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_48 2023-10-11T11:08:52.256 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_48.status 2023-10-11T11:08:52.256 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_48 --format xml 2023-10-11T11:08:52.256 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:08:52.301 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T11:08:52.302 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T11:08:52.303 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:-:1.1: Document is empty 2023-10-11T11:08:52.303 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: 2023-10-11T11:08:52.303 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:^ 2023-10-11T11:08:52.304 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id= 2023-10-11T11:08:52.304 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 213 = '' 2023-10-11T11:08:52.304 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T11:08:52.304 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.8 2023-10-11T11:08:53.105 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_48 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_48.status 2023-10-11T11:08:53.105 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:08:53.105 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:08:53.106 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_48 2023-10-11T11:08:53.106 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_48.status 2023-10-11T11:08:53.106 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_48 --format xml 2023-10-11T11:08:53.106 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:08:53.149 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T11:08:53.152 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id=213 2023-10-11T11:08:53.152 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 213 = 213 2023-10-11T11:08:53.152 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:08:53.152 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_snap_present cluster1 mirror image_48 snap 2023-10-11T11:08:53.152 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:08:53.153 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:08:53.153 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_48 2023-10-11T11:08:53.153 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap 2023-10-11T11:08:53.153 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T11:08:53.153 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-10-11T11:08:53.154 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 1 2023-10-11T11:08:54.152 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 info mirror/image_48@snap 2023-10-11T11:08:54.199 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:rbd image 'image_48': 2023-10-11T11:08:54.199 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: size 128 MiB in 32 objects 2023-10-11T11:08:54.200 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: order 22 (4 MiB objects) 2023-10-11T11:08:54.200 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: snapshot_count: 2 2023-10-11T11:08:54.200 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: id: 12898e64e354 2023-10-11T11:08:54.200 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: block_name_prefix: rbd_data.12898e64e354 2023-10-11T11:08:54.201 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: format: 2 2023-10-11T11:08:54.201 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-10-11T11:08:54.201 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: op_features: 2023-10-11T11:08:54.201 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: flags: 2023-10-11T11:08:54.202 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: create_timestamp: Wed Oct 11 11:01:51 2023 2023-10-11T11:08:54.202 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: access_timestamp: Wed Oct 11 11:01:51 2023 2023-10-11T11:08:54.202 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: modify_timestamp: Wed Oct 11 11:01:51 2023 2023-10-11T11:08:54.202 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: protected: False 2023-10-11T11:08:54.203 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring state: enabled 2023-10-11T11:08:54.203 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring mode: snapshot 2023-10-11T11:08:54.203 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring global id: a8b63bdf-0dc4-4ae7-bada-d821d5883be6 2023-10-11T11:08:54.203 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring primary: false 2023-10-11T11:08:54.205 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:08:54.205 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ compare_image_snaps mirror image_48 snap 2023-10-11T11:08:54.205 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:08:54.206 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_48 2023-10-11T11:08:54.206 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap 2023-10-11T11:08:54.206 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local ret=0 2023-10-11T11:08:54.206 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local rmt_export=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_48.export 2023-10-11T11:08:54.207 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local loc_export=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_48.export 2023-10-11T11:08:54.207 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rm -f /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_48.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_48.export 2023-10-11T11:08:54.207 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 -p mirror export image_48@snap /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_48.export 2023-10-11T11:08:54.378 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-10-11T11:08:54.383 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 -p mirror export image_48@snap /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_48.export 2023-10-11T11:08:54.630 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-10-11T11:08:54.636 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ cmp /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_48.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_48.export 2023-10-11T11:08:54.690 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rm -f /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_48.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_48.export 2023-10-11T11:08:54.728 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:08:54.728 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T11:08:54.728 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_49 2023-10-11T11:08:54.729 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ create_snap cluster2 mirror image_49 snap 2023-10-11T11:08:54.729 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:08:54.729 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:08:54.729 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_49 2023-10-11T11:08:54.730 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap 2023-10-11T11:08:54.730 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 -p mirror snap create image_49@snap --debug-rbd=20 --debug-journaler=20 2023-10-11T11:08:56.566 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_replay_started cluster1 mirror image_49 2023-10-11T11:08:56.566 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:08:56.567 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:08:56.567 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_49 2023-10-11T11:08:56.567 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_replay_state cluster1 mirror image_49 started 2023-10-11T11:08:56.567 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:08:56.567 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:08:56.568 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_49 2023-10-11T11:08:56.568 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local state=started 2023-10-11T11:08:56.568 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T11:08:56.568 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-10-11T11:08:56.568 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 1 2023-10-11T11:08:57.567 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test_image_replay_state cluster1 mirror image_49 started 2023-10-11T11:08:57.568 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:08:57.568 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:08:57.568 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_49 2023-10-11T11:08:57.568 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local test_state=started 2023-10-11T11:08:57.568 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local status_result 2023-10-11T11:08:57.569 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_state=stopped 2023-10-11T11:08:57.570 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_49 2023-10-11T11:08:57.570 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_instance=cluster1 2023-10-11T11:08:57.570 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T11:08:57.570 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ grep -i state 2023-10-11T11:08:57.570 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:08:57.571 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance=cluster1 2023-10-11T11:08:57.571 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local loop_instance 2023-10-11T11:08:57.571 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-10-11T11:08:57.571 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sleep 0 2023-10-11T11:08:57.571 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-10-11T11:08:57.572 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ seq 0 3 2023-10-11T11:08:57.573 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-10-11T11:08:57.574 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_49 2023-10-11T11:08:57.574 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1:0 2023-10-11T11:08:57.574 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T11:08:57.574 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance 2023-10-11T11:08:57.574 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-10-11T11:08:57.575 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local val=cluster1:0 2023-10-11T11:08:57.575 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_var_name=cluster 2023-10-11T11:08:57.575 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance_var_name=instance 2023-10-11T11:08:57.575 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ cluster=cluster1 2023-10-11T11:08:57.575 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ instance=0 2023-10-11T11:08:57.575 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' 0 = cluster1:0 ']' 2023-10-11T11:08:57.576 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval cluster=cluster1 2023-10-11T11:08:57.576 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:08:57.576 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval instance=0 2023-10-11T11:08:57.576 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=0 2023-10-11T11:08:57.576 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-10-11T11:08:57.577 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local local_cluster=cluster1:0 2023-10-11T11:08:57.578 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster=cluster1 2023-10-11T11:08:57.578 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance 2023-10-11T11:08:57.579 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-10-11T11:08:57.579 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local val=cluster1:0 2023-10-11T11:08:57.579 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster_var_name=local_cluster 2023-10-11T11:08:57.579 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance_var_name=instance 2023-10-11T11:08:57.579 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:08:57.579 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=0 2023-10-11T11:08:57.580 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ '[' 0 = cluster1:0 ']' 2023-10-11T11:08:57.580 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval local_cluster=cluster1 2023-10-11T11:08:57.580 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ local_cluster=cluster1 2023-10-11T11:08:57.580 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval instance=0 2023-10-11T11:08:57.580 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ instance=0 2023-10-11T11:08:57.581 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-10-11T11:08:57.594 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T11:08:57.595 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T11:08:57.595 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T11:08:57.595 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_49 2023-10-11T11:08:57.695 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:Invalid command: unused arguments: ['mirror/image_49'] 2023-10-11T11:08:57.695 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd mirror status : get status for rbd mirror 2023-10-11T11:08:57.695 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:admin_socket: invalid command 2023-10-11T11:08:57.698 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-10-11T11:08:57.699 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_49 2023-10-11T11:08:57.699 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1:1 2023-10-11T11:08:57.699 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T11:08:57.699 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance 2023-10-11T11:08:57.699 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-10-11T11:08:57.700 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local val=cluster1:1 2023-10-11T11:08:57.700 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_var_name=cluster 2023-10-11T11:08:57.700 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance_var_name=instance 2023-10-11T11:08:57.700 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ cluster=cluster1 2023-10-11T11:08:57.700 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ instance=1 2023-10-11T11:08:57.700 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' 1 = cluster1:1 ']' 2023-10-11T11:08:57.701 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval cluster=cluster1 2023-10-11T11:08:57.701 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:08:57.701 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval instance=1 2023-10-11T11:08:57.701 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=1 2023-10-11T11:08:57.701 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-10-11T11:08:57.702 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local local_cluster=cluster1:1 2023-10-11T11:08:57.702 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster=cluster1 2023-10-11T11:08:57.702 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance 2023-10-11T11:08:57.702 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-10-11T11:08:57.702 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local val=cluster1:1 2023-10-11T11:08:57.703 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster_var_name=local_cluster 2023-10-11T11:08:57.703 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance_var_name=instance 2023-10-11T11:08:57.703 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:08:57.703 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=1 2023-10-11T11:08:57.703 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ '[' 1 = cluster1:1 ']' 2023-10-11T11:08:57.703 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval local_cluster=cluster1 2023-10-11T11:08:57.704 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ local_cluster=cluster1 2023-10-11T11:08:57.704 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval instance=1 2023-10-11T11:08:57.704 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ instance=1 2023-10-11T11:08:57.704 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-10-11T11:08:57.717 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T11:08:57.717 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T11:08:57.717 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T11:08:57.718 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_49 2023-10-11T11:08:57.868 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ return 0 2023-10-11T11:08:57.869 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ status_result=' "state": "Replaying"' 2023-10-11T11:08:57.870 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ echo ' "state": "Replaying"' 2023-10-11T11:08:57.870 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ grep -i Replaying 2023-10-11T11:08:57.870 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: "state": "Replaying" 2023-10-11T11:08:57.872 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ current_state=started 2023-10-11T11:08:57.872 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test started = started 2023-10-11T11:08:57.872 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:08:57.872 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_49 2023-10-11T11:08:57.873 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local local_cluster=cluster1 2023-10-11T11:08:57.873 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:08:57.873 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:08:57.873 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_49 2023-10-11T11:08:57.873 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' snapshot = journal ']' 2023-10-11T11:08:57.874 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' snapshot = snapshot ']' 2023-10-11T11:08:57.874 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_49 2023-10-11T11:08:57.874 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local local_cluster=cluster1 2023-10-11T11:08:57.874 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:08:57.874 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:08:57.874 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_49 2023-10-11T11:08:57.875 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ mkfname cluster2-mirror-image_49.status 2023-10-11T11:08:57.875 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ echo cluster2-mirror-image_49.status 2023-10-11T11:08:57.875 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -e 's|[/ ]|_|g' 2023-10-11T11:08:57.876 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local status_log=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_49.status 2023-10-11T11:08:57.876 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ mkfname cluster1-mirror-image_49.status 2023-10-11T11:08:57.877 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ echo cluster1-mirror-image_49.status 2023-10-11T11:08:57.877 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -e 's|[/ ]|_|g' 2023-10-11T11:08:57.877 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local local_status_log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_49.status 2023-10-11T11:08:57.877 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ mirror_image_snapshot cluster2 mirror image_49 2023-10-11T11:08:57.878 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:08:57.878 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:08:57.878 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_49 2023-10-11T11:08:57.878 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_49 2023-10-11T11:08:58.567 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:Snapshot ID: 215 2023-10-11T11:08:58.579 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.5672390131459639) > weight (0.25) 2023-10-11T11:08:58.580 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.1 2023-10-11T11:08:58.580 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Sent signal 15 2023-10-11T11:08:58.580 INFO:tasks.rbd_mirror_thrash:waiting for 3 secs before reviving daemons 2023-10-11T11:08:58.581 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_49 /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_49.status 2023-10-11T11:08:58.581 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:08:58.581 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:08:58.582 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_49 2023-10-11T11:08:58.582 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_49.status 2023-10-11T11:08:58.582 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_49 --format xml 2023-10-11T11:08:58.582 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:08:58.582 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi111.stderr:2023-10-11T11:08:58.580+0000 7f651fafe700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 128795) UID: 1000 2023-10-11T11:08:58.635 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-10-11T11:08:58.637 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snapshot_id=215 2023-10-11T11:08:58.637 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T11:08:58.638 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T11:08:58.638 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.2 2023-10-11T11:08:58.838 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_49 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_49.status 2023-10-11T11:08:58.838 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:08:58.839 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:08:58.839 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_49 2023-10-11T11:08:58.839 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_49.status 2023-10-11T11:08:58.839 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_49 --format xml 2023-10-11T11:08:58.839 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:08:58.886 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T11:08:58.886 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T11:08:58.888 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:-:1.1: Document is empty 2023-10-11T11:08:58.888 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: 2023-10-11T11:08:58.888 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:^ 2023-10-11T11:08:58.889 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id= 2023-10-11T11:08:58.889 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 215 = '' 2023-10-11T11:08:58.889 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T11:08:58.889 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.4 2023-10-11T11:08:59.289 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_49 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_49.status 2023-10-11T11:08:59.290 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:08:59.290 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:08:59.290 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_49 2023-10-11T11:08:59.290 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_49.status 2023-10-11T11:08:59.291 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_49 --format xml 2023-10-11T11:08:59.291 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:08:59.341 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T11:08:59.342 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T11:08:59.343 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:-:1.1: Document is empty 2023-10-11T11:08:59.343 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: 2023-10-11T11:08:59.344 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:^ 2023-10-11T11:08:59.344 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id= 2023-10-11T11:08:59.344 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 215 = '' 2023-10-11T11:08:59.344 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T11:08:59.345 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.8 2023-10-11T11:09:00.146 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_49 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_49.status 2023-10-11T11:09:00.146 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:09:00.146 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:09:00.147 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_49 2023-10-11T11:09:00.147 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_49.status 2023-10-11T11:09:00.147 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_49 --format xml 2023-10-11T11:09:00.147 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:09:00.190 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T11:09:00.192 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id=215 2023-10-11T11:09:00.193 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 215 = 215 2023-10-11T11:09:00.193 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:09:00.193 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_snap_present cluster1 mirror image_49 snap 2023-10-11T11:09:00.193 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:09:00.193 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:09:00.193 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_49 2023-10-11T11:09:00.194 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap 2023-10-11T11:09:00.194 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T11:09:00.194 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-10-11T11:09:00.194 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 1 2023-10-11T11:09:01.193 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 info mirror/image_49@snap 2023-10-11T11:09:01.239 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:rbd image 'image_49': 2023-10-11T11:09:01.239 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: size 128 MiB in 32 objects 2023-10-11T11:09:01.239 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: order 22 (4 MiB objects) 2023-10-11T11:09:01.240 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: snapshot_count: 2 2023-10-11T11:09:01.240 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: id: 12894b07a2d7 2023-10-11T11:09:01.240 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: block_name_prefix: rbd_data.12894b07a2d7 2023-10-11T11:09:01.240 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: format: 2 2023-10-11T11:09:01.241 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-10-11T11:09:01.241 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: op_features: 2023-10-11T11:09:01.241 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: flags: 2023-10-11T11:09:01.241 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: create_timestamp: Wed Oct 11 11:02:02 2023 2023-10-11T11:09:01.241 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: access_timestamp: Wed Oct 11 11:02:02 2023 2023-10-11T11:09:01.242 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: modify_timestamp: Wed Oct 11 11:02:02 2023 2023-10-11T11:09:01.242 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: protected: False 2023-10-11T11:09:01.242 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring state: enabled 2023-10-11T11:09:01.242 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring mode: snapshot 2023-10-11T11:09:01.242 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring global id: 80e3527d-c1f3-41a7-917b-e93369287a62 2023-10-11T11:09:01.243 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring primary: false 2023-10-11T11:09:01.244 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:09:01.244 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ compare_image_snaps mirror image_49 snap 2023-10-11T11:09:01.244 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:09:01.244 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_49 2023-10-11T11:09:01.245 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap 2023-10-11T11:09:01.245 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local ret=0 2023-10-11T11:09:01.245 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local rmt_export=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_49.export 2023-10-11T11:09:01.245 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local loc_export=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_49.export 2023-10-11T11:09:01.245 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rm -f /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_49.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_49.export 2023-10-11T11:09:01.246 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 -p mirror export image_49@snap /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_49.export 2023-10-11T11:09:01.423 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-10-11T11:09:01.430 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 -p mirror export image_49@snap /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_49.export 2023-10-11T11:09:01.581 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.1 2023-10-11T11:09:01.582 INFO:teuthology.orchestra.run:waiting for 600 2023-10-11T11:09:01.582 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.1 2023-10-11T11:09:01.582 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2023-10-11T11:09:01.583 DEBUG:teuthology.orchestra.run.smithi111:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2023-10-11T11:09:01.585 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2023-10-11T11:09:01.585 INFO:tasks.rbd_mirror_thrash:waiting for 97 secs before thrashing 2023-10-11T11:09:01.633 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-10-11T11:09:01.640 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ cmp /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_49.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_49.export 2023-10-11T11:09:01.697 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rm -f /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_49.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_49.export 2023-10-11T11:09:01.736 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:09:01.736 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T11:09:01.737 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_50 2023-10-11T11:09:01.737 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ create_snap cluster2 mirror image_50 snap 2023-10-11T11:09:01.737 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:09:01.737 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:09:01.737 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_50 2023-10-11T11:09:01.738 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap 2023-10-11T11:09:01.738 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 -p mirror snap create image_50@snap --debug-rbd=20 --debug-journaler=20 2023-10-11T11:09:03.585 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_replay_started cluster1 mirror image_50 2023-10-11T11:09:03.586 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:09:03.586 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:09:03.586 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_50 2023-10-11T11:09:03.586 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_replay_state cluster1 mirror image_50 started 2023-10-11T11:09:03.587 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:09:03.587 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:09:03.587 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_50 2023-10-11T11:09:03.587 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local state=started 2023-10-11T11:09:03.588 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T11:09:03.588 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-10-11T11:09:03.588 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 1 2023-10-11T11:09:04.587 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test_image_replay_state cluster1 mirror image_50 started 2023-10-11T11:09:04.588 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:09:04.588 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:09:04.589 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_50 2023-10-11T11:09:04.589 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local test_state=started 2023-10-11T11:09:04.589 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local status_result 2023-10-11T11:09:04.589 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_state=stopped 2023-10-11T11:09:04.590 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_50 2023-10-11T11:09:04.590 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_instance=cluster1 2023-10-11T11:09:04.590 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ grep -i state 2023-10-11T11:09:04.591 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T11:09:04.591 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:09:04.591 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance=cluster1 2023-10-11T11:09:04.591 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local loop_instance 2023-10-11T11:09:04.592 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-10-11T11:09:04.592 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sleep 0 2023-10-11T11:09:04.596 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-10-11T11:09:04.596 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ seq 0 3 2023-10-11T11:09:04.596 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-10-11T11:09:04.597 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_50 2023-10-11T11:09:04.597 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1:0 2023-10-11T11:09:04.597 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T11:09:04.598 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance 2023-10-11T11:09:04.598 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-10-11T11:09:04.598 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local val=cluster1:0 2023-10-11T11:09:04.598 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_var_name=cluster 2023-10-11T11:09:04.599 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance_var_name=instance 2023-10-11T11:09:04.599 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ cluster=cluster1 2023-10-11T11:09:04.599 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ instance=0 2023-10-11T11:09:04.599 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' 0 = cluster1:0 ']' 2023-10-11T11:09:04.600 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval cluster=cluster1 2023-10-11T11:09:04.600 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:09:04.600 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval instance=0 2023-10-11T11:09:04.601 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=0 2023-10-11T11:09:04.601 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-10-11T11:09:04.601 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local local_cluster=cluster1:0 2023-10-11T11:09:04.601 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster=cluster1 2023-10-11T11:09:04.602 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance 2023-10-11T11:09:04.602 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-10-11T11:09:04.602 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local val=cluster1:0 2023-10-11T11:09:04.602 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster_var_name=local_cluster 2023-10-11T11:09:04.603 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance_var_name=instance 2023-10-11T11:09:04.603 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:09:04.603 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=0 2023-10-11T11:09:04.603 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ '[' 0 = cluster1:0 ']' 2023-10-11T11:09:04.604 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval local_cluster=cluster1 2023-10-11T11:09:04.604 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ local_cluster=cluster1 2023-10-11T11:09:04.604 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval instance=0 2023-10-11T11:09:04.604 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ instance=0 2023-10-11T11:09:04.605 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-10-11T11:09:04.613 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T11:09:04.613 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T11:09:04.614 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-10-11T11:09:04.614 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_50 2023-10-11T11:09:04.716 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:Invalid command: unused arguments: ['mirror/image_50'] 2023-10-11T11:09:04.716 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd mirror status : get status for rbd mirror 2023-10-11T11:09:04.716 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:admin_socket: invalid command 2023-10-11T11:09:04.719 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-10-11T11:09:04.720 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_50 2023-10-11T11:09:04.720 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1:1 2023-10-11T11:09:04.720 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ shift 2023-10-11T11:09:04.720 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance 2023-10-11T11:09:04.720 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-10-11T11:09:04.721 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local val=cluster1:1 2023-10-11T11:09:04.721 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster_var_name=cluster 2023-10-11T11:09:04.721 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local instance_var_name=instance 2023-10-11T11:09:04.721 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ cluster=cluster1 2023-10-11T11:09:04.721 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ instance=1 2023-10-11T11:09:04.722 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ '[' 1 = cluster1:1 ']' 2023-10-11T11:09:04.722 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval cluster=cluster1 2023-10-11T11:09:04.722 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:09:04.722 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ eval instance=1 2023-10-11T11:09:04.722 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=1 2023-10-11T11:09:04.723 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-10-11T11:09:04.723 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local local_cluster=cluster1:1 2023-10-11T11:09:04.723 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster=cluster1 2023-10-11T11:09:04.723 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance 2023-10-11T11:09:04.723 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-10-11T11:09:04.724 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local val=cluster1:1 2023-10-11T11:09:04.724 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local cluster_var_name=local_cluster 2023-10-11T11:09:04.724 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ local instance_var_name=instance 2023-10-11T11:09:04.724 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ cluster=cluster1 2023-10-11T11:09:04.724 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ instance=1 2023-10-11T11:09:04.725 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ '[' 1 = cluster1:1 ']' 2023-10-11T11:09:04.725 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval local_cluster=cluster1 2023-10-11T11:09:04.725 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ local_cluster=cluster1 2023-10-11T11:09:04.725 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ eval instance=1 2023-10-11T11:09:04.725 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ instance=1 2023-10-11T11:09:04.726 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-10-11T11:09:04.738 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T11:09:04.738 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T11:09:04.738 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-10-11T11:09:04.738 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_50 2023-10-11T11:09:04.895 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ return 0 2023-10-11T11:09:04.895 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ status_result=' "state": "Replaying"' 2023-10-11T11:09:04.896 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ echo ' "state": "Replaying"' 2023-10-11T11:09:04.896 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ grep -i Replaying 2023-10-11T11:09:04.896 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: "state": "Replaying" 2023-10-11T11:09:04.898 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ current_state=started 2023-10-11T11:09:04.898 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test started = started 2023-10-11T11:09:04.898 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:09:04.898 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_50 2023-10-11T11:09:04.899 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local local_cluster=cluster1 2023-10-11T11:09:04.899 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:09:04.899 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:09:04.899 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_50 2023-10-11T11:09:04.899 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' snapshot = journal ']' 2023-10-11T11:09:04.900 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' snapshot = snapshot ']' 2023-10-11T11:09:04.900 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_50 2023-10-11T11:09:04.900 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local local_cluster=cluster1 2023-10-11T11:09:04.900 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:09:04.900 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:09:04.900 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_50 2023-10-11T11:09:04.901 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ mkfname cluster2-mirror-image_50.status 2023-10-11T11:09:04.901 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ echo cluster2-mirror-image_50.status 2023-10-11T11:09:04.901 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -e 's|[/ ]|_|g' 2023-10-11T11:09:04.902 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local status_log=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_50.status 2023-10-11T11:09:04.902 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ mkfname cluster1-mirror-image_50.status 2023-10-11T11:09:04.902 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ echo cluster1-mirror-image_50.status 2023-10-11T11:09:04.903 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -e 's|[/ ]|_|g' 2023-10-11T11:09:04.903 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local local_status_log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_50.status 2023-10-11T11:09:04.903 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ mirror_image_snapshot cluster2 mirror image_50 2023-10-11T11:09:04.903 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:09:04.903 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:09:04.903 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_50 2023-10-11T11:09:04.904 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_50 2023-10-11T11:09:05.584 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:Snapshot ID: 217 2023-10-11T11:09:05.595 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_50 /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_50.status 2023-10-11T11:09:05.595 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:09:05.595 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:09:05.595 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_50 2023-10-11T11:09:05.596 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_50.status 2023-10-11T11:09:05.596 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_50 --format xml 2023-10-11T11:09:05.596 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:09:05.636 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-10-11T11:09:05.639 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snapshot_id=217 2023-10-11T11:09:05.639 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T11:09:05.639 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T11:09:05.639 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.2 2023-10-11T11:09:05.840 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_50 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_50.status 2023-10-11T11:09:05.840 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:09:05.840 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:09:05.841 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_50 2023-10-11T11:09:05.841 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_50.status 2023-10-11T11:09:05.841 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_50 --format xml 2023-10-11T11:09:05.841 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:09:05.886 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T11:09:05.888 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id=117 2023-10-11T11:09:05.888 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 217 = 117 2023-10-11T11:09:05.888 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T11:09:05.888 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.4 2023-10-11T11:09:06.289 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_50 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_50.status 2023-10-11T11:09:06.289 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:09:06.289 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:09:06.289 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_50 2023-10-11T11:09:06.290 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_50.status 2023-10-11T11:09:06.290 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_50 --format xml 2023-10-11T11:09:06.290 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:09:06.336 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T11:09:06.336 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T11:09:06.337 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:-:1.1: Document is empty 2023-10-11T11:09:06.337 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: 2023-10-11T11:09:06.338 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:^ 2023-10-11T11:09:06.338 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id= 2023-10-11T11:09:06.338 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 217 = '' 2023-10-11T11:09:06.339 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T11:09:06.339 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.8 2023-10-11T11:09:07.139 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_50 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_50.status 2023-10-11T11:09:07.139 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:09:07.140 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:09:07.140 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_50 2023-10-11T11:09:07.140 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_50.status 2023-10-11T11:09:07.140 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_50 --format xml 2023-10-11T11:09:07.140 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:09:07.210 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T11:09:07.210 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T11:09:07.211 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:-:1.1: Document is empty 2023-10-11T11:09:07.211 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: 2023-10-11T11:09:07.211 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:^ 2023-10-11T11:09:07.212 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id= 2023-10-11T11:09:07.212 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 217 = '' 2023-10-11T11:09:07.213 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-10-11T11:09:07.213 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 1.6 2023-10-11T11:09:08.813 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_50 /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_50.status 2023-10-11T11:09:08.814 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:09:08.814 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:09:08.815 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_50 2023-10-11T11:09:08.815 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local log=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_50.status 2023-10-11T11:09:08.815 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_50 --format xml 2023-10-11T11:09:08.815 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-10-11T11:09:08.865 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-10-11T11:09:08.868 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local primary_snapshot_id=217 2023-10-11T11:09:08.868 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 217 = 217 2023-10-11T11:09:08.868 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:09:08.869 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_snap_present cluster1 mirror image_50 snap 2023-10-11T11:09:08.869 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:09:08.869 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:09:08.870 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_50 2023-10-11T11:09:08.870 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap 2023-10-11T11:09:08.870 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T11:09:08.870 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-10-11T11:09:08.871 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 1 2023-10-11T11:09:09.869 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 info mirror/image_50@snap 2023-10-11T11:09:09.914 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:rbd image 'image_50': 2023-10-11T11:09:09.915 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: size 128 MiB in 32 objects 2023-10-11T11:09:09.915 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: order 22 (4 MiB objects) 2023-10-11T11:09:09.915 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: snapshot_count: 2 2023-10-11T11:09:09.915 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: id: 12894a703a66 2023-10-11T11:09:09.915 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: block_name_prefix: rbd_data.12894a703a66 2023-10-11T11:09:09.916 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: format: 2 2023-10-11T11:09:09.916 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-10-11T11:09:09.916 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: op_features: 2023-10-11T11:09:09.916 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: flags: 2023-10-11T11:09:09.917 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: create_timestamp: Wed Oct 11 11:02:11 2023 2023-10-11T11:09:09.917 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: access_timestamp: Wed Oct 11 11:02:11 2023 2023-10-11T11:09:09.917 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: modify_timestamp: Wed Oct 11 11:02:11 2023 2023-10-11T11:09:09.917 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: protected: False 2023-10-11T11:09:09.917 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring state: enabled 2023-10-11T11:09:09.918 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring mode: snapshot 2023-10-11T11:09:09.918 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring global id: cd60b48a-16a6-4a4e-8bc0-70e977b76bac 2023-10-11T11:09:09.918 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring primary: false 2023-10-11T11:09:09.919 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:09:09.920 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ compare_image_snaps mirror image_50 snap 2023-10-11T11:09:09.920 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:09:09.920 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_50 2023-10-11T11:09:09.920 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap_name=snap 2023-10-11T11:09:09.920 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local ret=0 2023-10-11T11:09:09.921 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local rmt_export=/tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_50.export 2023-10-11T11:09:09.921 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local loc_export=/tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_50.export 2023-10-11T11:09:09.921 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rm -f /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_50.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_50.export 2023-10-11T11:09:09.921 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 -p mirror export image_50@snap /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_50.export 2023-10-11T11:09:10.098 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-10-11T11:09:10.104 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster1 -p mirror export image_50@snap /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_50.export 2023-10-11T11:09:10.330 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-10-11T11:09:10.337 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ cmp /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_50.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_50.export 2023-10-11T11:09:10.392 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rm -f /tmp/tmp.nGgfBEp0YO/cluster2-mirror-image_50.export /tmp/tmp.nGgfBEp0YO/cluster1-mirror-image_50.export 2023-10-11T11:09:10.428 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:09:10.428 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ testlog 'TEST: delete many images' 2023-10-11T11:09:10.429 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ tee -a /tmp/tmp.nGgfBEp0YO/rbd-mirror.test.log 2023-10-11T11:09:10.429 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ date '+%F %T' 2023-10-11T11:09:10.430 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ echo 2023-10-11 11:09:10 TEST: delete many images 2023-10-11T11:09:10.430 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:2023-10-11 11:09:10 TEST: delete many images 2023-10-11T11:09:10.430 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ seq 1 50 2023-10-11T11:09:10.431 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T11:09:10.431 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_1 2023-10-11T11:09:10.432 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_snapshot cluster2 mirror image_1 snap 2023-10-11T11:09:10.432 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:09:10.432 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:09:10.432 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_1 2023-10-11T11:09:10.432 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap=snap 2023-10-11T11:09:10.432 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 snap rm mirror/image_1@snap 2023-10-11T11:09:11.428 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Removing snap: 100% complete...done. 2023-10-11T11:09:11.448 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_image_retry cluster2 mirror image_1 2023-10-11T11:09:11.448 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:09:11.449 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:09:11.449 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_1 2023-10-11T11:09:11.449 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0 1 2 4 8 16 32 2023-10-11T11:09:11.449 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0 2023-10-11T11:09:11.450 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_image cluster2 mirror image_1 2023-10-11T11:09:11.450 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:09:11.450 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:09:11.450 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_1 2023-10-11T11:09:11.451 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_1 2023-10-11T11:09:11.493 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 rm mirror/image_1 2023-10-11T11:09:11.618 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi111.stderr:2023-10-11T11:09:11.617+0000 7f1e687ca700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x556984c869c0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-10-11T11:09:11.618 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi111.stderr:2023-10-11T11:09:11.617+0000 7f1e6f7d8700 -1 rbd::mirror::ImageReplayer: 0x556984a6cc80 [3/eeb747e2-80b4-485a-95e6-a07d67abb191] operator(): start failed: (85) Interrupted system call should be restarted 2023-10-11T11:09:13.568 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-10-11T11:09:13.572 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:09:13.572 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T11:09:13.572 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_2 2023-10-11T11:09:13.572 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_snapshot cluster2 mirror image_2 snap 2023-10-11T11:09:13.573 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:09:13.573 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:09:13.573 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_2 2023-10-11T11:09:13.573 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap=snap 2023-10-11T11:09:13.573 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 snap rm mirror/image_2@snap 2023-10-11T11:09:14.436 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Removing snap: 100% complete...done. 2023-10-11T11:09:14.454 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_image_retry cluster2 mirror image_2 2023-10-11T11:09:14.454 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:09:14.454 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:09:14.454 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_2 2023-10-11T11:09:14.455 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0 1 2 4 8 16 32 2023-10-11T11:09:14.455 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0 2023-10-11T11:09:14.455 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_image cluster2 mirror image_2 2023-10-11T11:09:14.456 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:09:14.456 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:09:14.456 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_2 2023-10-11T11:09:14.456 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_2 2023-10-11T11:09:14.497 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 rm mirror/image_2 2023-10-11T11:09:14.624 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi111.stderr:2023-10-11T11:09:14.622+0000 7f1e68fcb700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x556984cc6b60 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-10-11T11:09:14.624 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi111.stderr:2023-10-11T11:09:14.622+0000 7f1e6efd7700 -1 rbd::mirror::ImageReplayer: 0x556983bd1400 [3/115d7fef-7d84-4b5a-aaf4-a5ee31ecb254] operator(): start failed: (85) Interrupted system call should be restarted 2023-10-11T11:09:16.587 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-10-11T11:09:16.590 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:09:16.591 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T11:09:16.591 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_3 2023-10-11T11:09:16.591 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_snapshot cluster2 mirror image_3 snap 2023-10-11T11:09:16.591 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:09:16.591 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:09:16.592 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_3 2023-10-11T11:09:16.592 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap=snap 2023-10-11T11:09:16.592 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 snap rm mirror/image_3@snap 2023-10-11T11:09:17.473 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Removing snap: 100% complete...done. 2023-10-11T11:09:17.488 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_image_retry cluster2 mirror image_3 2023-10-11T11:09:17.489 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:09:17.489 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:09:17.489 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_3 2023-10-11T11:09:17.489 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0 1 2 4 8 16 32 2023-10-11T11:09:17.489 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0 2023-10-11T11:09:17.490 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_image cluster2 mirror image_3 2023-10-11T11:09:17.490 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:09:17.490 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:09:17.490 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_3 2023-10-11T11:09:17.491 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_3 2023-10-11T11:09:17.537 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 rm mirror/image_3 2023-10-11T11:09:17.818 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi111.stderr:2023-10-11T11:09:17.817+0000 7f2c36eed700 -1 rbd::mirror::ImageReplayer: 0x55c1e6531180 [3/665a8ec8-563e-4515-ba87-41db0aac57ac] start: already running 2023-10-11T11:09:17.819 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi111.stderr:2023-10-11T11:09:17.817+0000 7f2c36eed700 -1 rbd::mirror::ImageReplayer: 0x55c1e6531180 [3/665a8ec8-563e-4515-ba87-41db0aac57ac] start: already running 2023-10-11T11:09:17.851 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi111.stderr:2023-10-11T11:09:17.851+0000 7f1e687ca700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x556984cc6d00 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-10-11T11:09:17.852 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi111.stderr:2023-10-11T11:09:17.851+0000 7f1e6efd7700 -1 rbd::mirror::ImageReplayer: 0x5569846da280 [3/665a8ec8-563e-4515-ba87-41db0aac57ac] operator(): start failed: (85) Interrupted system call should be restarted 2023-10-11T11:09:19.604 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-10-11T11:09:19.608 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:09:19.608 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T11:09:19.608 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_4 2023-10-11T11:09:19.609 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_snapshot cluster2 mirror image_4 snap 2023-10-11T11:09:19.609 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:09:19.609 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:09:19.609 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_4 2023-10-11T11:09:19.610 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap=snap 2023-10-11T11:09:19.610 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 snap rm mirror/image_4@snap 2023-10-11T11:09:20.431 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Removing snap: 100% complete...done. 2023-10-11T11:09:20.446 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_image_retry cluster2 mirror image_4 2023-10-11T11:09:20.446 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:09:20.447 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:09:20.447 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_4 2023-10-11T11:09:20.447 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0 1 2 4 8 16 32 2023-10-11T11:09:20.447 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0 2023-10-11T11:09:20.448 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_image cluster2 mirror image_4 2023-10-11T11:09:20.449 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:09:20.449 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:09:20.449 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_4 2023-10-11T11:09:20.449 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_4 2023-10-11T11:09:20.699 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 rm mirror/image_4 2023-10-11T11:09:20.815 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi111.stderr:2023-10-11T11:09:20.814+0000 7f1e687ca700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x556984c86b60 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-10-11T11:09:20.815 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi111.stderr:2023-10-11T11:09:20.814+0000 7f1e6f7d8700 -1 rbd::mirror::ImageReplayer: 0x556984a6cf00 [3/f031088c-5560-4c60-b77f-f68d8adaedd0] operator(): start failed: (85) Interrupted system call should be restarted 2023-10-11T11:09:22.609 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-10-11T11:09:22.613 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:09:22.613 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T11:09:22.613 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_5 2023-10-11T11:09:22.613 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_snapshot cluster2 mirror image_5 snap 2023-10-11T11:09:22.613 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:09:22.614 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:09:22.614 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_5 2023-10-11T11:09:22.614 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap=snap 2023-10-11T11:09:22.614 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 snap rm mirror/image_5@snap 2023-10-11T11:09:23.480 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Removing snap: 100% complete...done. 2023-10-11T11:09:23.501 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_image_retry cluster2 mirror image_5 2023-10-11T11:09:23.501 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:09:23.501 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:09:23.501 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_5 2023-10-11T11:09:23.502 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0 1 2 4 8 16 32 2023-10-11T11:09:23.502 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0 2023-10-11T11:09:23.502 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_image cluster2 mirror image_5 2023-10-11T11:09:23.503 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:09:23.503 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:09:23.503 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_5 2023-10-11T11:09:23.503 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_5 2023-10-11T11:09:23.547 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 rm mirror/image_5 2023-10-11T11:09:23.662 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi111.stderr:2023-10-11T11:09:23.661+0000 7f1e68fcb700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x556984c86d00 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-10-11T11:09:23.663 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi111.stderr:2023-10-11T11:09:23.661+0000 7f1e6f7d8700 -1 rbd::mirror::ImageReplayer: 0x556984a07400 [3/a9ff47ee-c486-4ecb-981d-63d0dafd14e5] operator(): start failed: (85) Interrupted system call should be restarted 2023-10-11T11:09:25.556 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-10-11T11:09:25.560 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:09:25.560 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T11:09:25.560 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_6 2023-10-11T11:09:25.560 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_snapshot cluster2 mirror image_6 snap 2023-10-11T11:09:25.561 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:09:25.561 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:09:25.561 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_6 2023-10-11T11:09:25.561 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap=snap 2023-10-11T11:09:25.561 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 snap rm mirror/image_6@snap 2023-10-11T11:09:26.486 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Removing snap: 100% complete...done. 2023-10-11T11:09:26.516 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_image_retry cluster2 mirror image_6 2023-10-11T11:09:26.516 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:09:26.516 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:09:26.516 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_6 2023-10-11T11:09:26.517 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0 1 2 4 8 16 32 2023-10-11T11:09:26.517 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0 2023-10-11T11:09:26.518 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_image cluster2 mirror image_6 2023-10-11T11:09:26.518 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:09:26.519 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:09:26.519 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_6 2023-10-11T11:09:26.519 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_6 2023-10-11T11:09:26.564 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 rm mirror/image_6 2023-10-11T11:09:26.683 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi111.stderr:2023-10-11T11:09:26.682+0000 7f1e687ca700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x556984cc6ea0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-10-11T11:09:26.684 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi111.stderr:2023-10-11T11:09:26.682+0000 7f1e6efd7700 -1 rbd::mirror::ImageReplayer: 0x5569846e5400 [3/7ce830b7-0256-4d8e-905d-503bebab5873] operator(): start failed: (85) Interrupted system call should be restarted 2023-10-11T11:09:28.632 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-10-11T11:09:28.636 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:09:28.636 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T11:09:28.637 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_7 2023-10-11T11:09:28.637 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_snapshot cluster2 mirror image_7 snap 2023-10-11T11:09:28.637 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:09:28.637 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:09:28.638 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_7 2023-10-11T11:09:28.638 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap=snap 2023-10-11T11:09:28.638 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 snap rm mirror/image_7@snap 2023-10-11T11:09:29.507 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Removing snap: 100% complete...done. 2023-10-11T11:09:29.522 INFO:tasks.ceph.cluster2.osd.2.smithi111.stderr:2023-10-11T11:09:29.521+0000 7fa35bd9b700 -1 reset not still connected to 0x556437861ba0 2023-10-11T11:09:29.523 INFO:tasks.ceph.cluster2.osd.2.smithi111.stderr:2023-10-11T11:09:29.521+0000 7fa35bd9b700 -1 reset not still connected to 0x556439b8f2b0 2023-10-11T11:09:29.523 INFO:tasks.ceph.cluster2.osd.2.smithi111.stderr:2023-10-11T11:09:29.521+0000 7fa35bd9b700 -1 reset not still connected to 0x55643b454d00 2023-10-11T11:09:29.527 INFO:tasks.ceph.cluster2.osd.2.smithi111.stderr:2023-10-11T11:09:29.526+0000 7fa35bd9b700 -1 reset not still connected to 0x55643f0a4680 2023-10-11T11:09:29.527 INFO:tasks.ceph.cluster2.osd.2.smithi111.stderr:2023-10-11T11:09:29.526+0000 7fa35bd9b700 -1 reset not still connected to 0x55644081c4e0 2023-10-11T11:09:29.527 INFO:tasks.ceph.cluster2.osd.2.smithi111.stderr:2023-10-11T11:09:29.526+0000 7fa35bd9b700 -1 reset not still connected to 0x55644081d040 2023-10-11T11:09:29.539 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_image_retry cluster2 mirror image_7 2023-10-11T11:09:29.540 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:09:29.540 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:09:29.540 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_7 2023-10-11T11:09:29.541 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0 1 2 4 8 16 32 2023-10-11T11:09:29.541 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0 2023-10-11T11:09:29.542 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_image cluster2 mirror image_7 2023-10-11T11:09:29.542 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:09:29.542 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:09:29.543 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_7 2023-10-11T11:09:29.543 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_7 2023-10-11T11:09:29.584 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 rm mirror/image_7 2023-10-11T11:09:29.725 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi111.stderr:2023-10-11T11:09:29.725+0000 7f1e68fcb700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x556984c86680 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-10-11T11:09:29.742 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi111.stderr:2023-10-11T11:09:29.741+0000 7f1e68fcb700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x556984373380 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-10-11T11:09:29.742 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi111.stderr:2023-10-11T11:09:29.741+0000 7f1e6f7d8700 -1 rbd::mirror::ImageReplayer: 0x556984a06a00 [3/96782a85-6dfc-4ac8-9a0a-1c21292c349e] operator(): start failed: (85) Interrupted system call should be restarted 2023-10-11T11:09:31.453 INFO:tasks.ceph.cluster2.osd.2.smithi111.stderr:2023-10-11T11:09:31.452+0000 7fa35bd9b700 -1 reset not still connected to 0x5564219388f0 2023-10-11T11:09:31.687 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-10-11T11:09:31.690 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:09:31.691 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T11:09:31.691 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_8 2023-10-11T11:09:31.691 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_snapshot cluster2 mirror image_8 snap 2023-10-11T11:09:31.692 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:09:31.692 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:09:31.692 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_8 2023-10-11T11:09:31.692 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap=snap 2023-10-11T11:09:31.693 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 snap rm mirror/image_8@snap 2023-10-11T11:09:32.510 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Removing snap: 100% complete...done. 2023-10-11T11:09:32.526 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_image_retry cluster2 mirror image_8 2023-10-11T11:09:32.526 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:09:32.527 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:09:32.527 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_8 2023-10-11T11:09:32.527 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0 1 2 4 8 16 32 2023-10-11T11:09:32.527 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0 2023-10-11T11:09:32.528 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_image cluster2 mirror image_8 2023-10-11T11:09:32.529 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:09:32.529 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:09:32.529 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_8 2023-10-11T11:09:32.529 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_8 2023-10-11T11:09:32.574 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 rm mirror/image_8 2023-10-11T11:09:32.695 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi111.stderr:2023-10-11T11:09:32.694+0000 7f1e68fcb700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x556984cc71e0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-10-11T11:09:32.696 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi111.stderr:2023-10-11T11:09:32.694+0000 7f1e6efd7700 -1 rbd::mirror::ImageReplayer: 0x5569846e4500 [3/ff8c48f9-2b0b-4783-bc85-bfb3318837fb] operator(): start failed: (85) Interrupted system call should be restarted 2023-10-11T11:09:34.650 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-10-11T11:09:34.654 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:09:34.654 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T11:09:34.654 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_9 2023-10-11T11:09:34.654 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_snapshot cluster2 mirror image_9 snap 2023-10-11T11:09:34.655 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:09:34.655 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:09:34.655 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_9 2023-10-11T11:09:34.655 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap=snap 2023-10-11T11:09:34.655 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 snap rm mirror/image_9@snap 2023-10-11T11:09:35.430 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Removing snap: 100% complete...done. 2023-10-11T11:09:35.460 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_image_retry cluster2 mirror image_9 2023-10-11T11:09:35.461 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:09:35.461 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:09:35.461 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_9 2023-10-11T11:09:35.461 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0 1 2 4 8 16 32 2023-10-11T11:09:35.461 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0 2023-10-11T11:09:35.462 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_image cluster2 mirror image_9 2023-10-11T11:09:35.462 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:09:35.462 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:09:35.463 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_9 2023-10-11T11:09:35.463 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_9 2023-10-11T11:09:35.504 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 rm mirror/image_9 2023-10-11T11:09:35.621 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi111.stderr:2023-10-11T11:09:35.620+0000 7f1e68fcb700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x556984c77d40 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-10-11T11:09:35.621 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi111.stderr:2023-10-11T11:09:35.620+0000 7f1e6f7d8700 -1 rbd::mirror::ImageReplayer: 0x556984a07b80 [3/ca5ef4e0-7cfe-484d-8d61-93b7e745a919] operator(): start failed: (85) Interrupted system call should be restarted 2023-10-11T11:09:37.660 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-10-11T11:09:37.664 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:09:37.664 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T11:09:37.665 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_10 2023-10-11T11:09:37.665 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_snapshot cluster2 mirror image_10 snap 2023-10-11T11:09:37.665 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:09:37.665 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:09:37.665 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_10 2023-10-11T11:09:37.665 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap=snap 2023-10-11T11:09:37.666 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 snap rm mirror/image_10@snap 2023-10-11T11:09:38.534 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Removing snap: 100% complete...done. 2023-10-11T11:09:38.566 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_image_retry cluster2 mirror image_10 2023-10-11T11:09:38.566 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:09:38.566 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:09:38.566 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_10 2023-10-11T11:09:38.567 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0 1 2 4 8 16 32 2023-10-11T11:09:38.567 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0 2023-10-11T11:09:38.567 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_image cluster2 mirror image_10 2023-10-11T11:09:38.568 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:09:38.568 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:09:38.568 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_10 2023-10-11T11:09:38.568 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_10 2023-10-11T11:09:38.613 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 rm mirror/image_10 2023-10-11T11:09:38.731 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi111.stderr:2023-10-11T11:09:38.730+0000 7f1e687ca700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x556984cc7380 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-10-11T11:09:38.731 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi111.stderr:2023-10-11T11:09:38.730+0000 7f1e6efd7700 -1 rbd::mirror::ImageReplayer: 0x556984a06780 [3/965cf2d0-179c-4721-be88-950f8a10b7ae] operator(): start failed: (85) Interrupted system call should be restarted 2023-10-11T11:09:40.566 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-10-11T11:09:40.570 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:09:40.570 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T11:09:40.570 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_11 2023-10-11T11:09:40.571 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_snapshot cluster2 mirror image_11 snap 2023-10-11T11:09:40.571 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:09:40.571 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:09:40.571 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_11 2023-10-11T11:09:40.571 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap=snap 2023-10-11T11:09:40.572 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 snap rm mirror/image_11@snap 2023-10-11T11:09:41.543 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Removing snap: 100% complete...done. 2023-10-11T11:09:41.564 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_image_retry cluster2 mirror image_11 2023-10-11T11:09:41.565 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:09:41.565 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:09:41.565 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_11 2023-10-11T11:09:41.565 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0 1 2 4 8 16 32 2023-10-11T11:09:41.566 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0 2023-10-11T11:09:41.566 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_image cluster2 mirror image_11 2023-10-11T11:09:41.567 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:09:41.567 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:09:41.567 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_11 2023-10-11T11:09:41.567 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_11 2023-10-11T11:09:41.619 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 rm mirror/image_11 2023-10-11T11:09:41.746 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi111.stderr:2023-10-11T11:09:41.745+0000 7f1e687ca700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x556984cc7520 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-10-11T11:09:41.747 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi111.stderr:2023-10-11T11:09:41.745+0000 7f1e6efd7700 -1 rbd::mirror::ImageReplayer: 0x5569846db180 [3/35668704-fbeb-4c06-998e-79dc6d9e7e60] operator(): start failed: (85) Interrupted system call should be restarted 2023-10-11T11:09:43.685 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-10-11T11:09:43.689 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:09:43.689 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T11:09:43.689 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_12 2023-10-11T11:09:43.689 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_snapshot cluster2 mirror image_12 snap 2023-10-11T11:09:43.689 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:09:43.690 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:09:43.690 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_12 2023-10-11T11:09:43.690 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap=snap 2023-10-11T11:09:43.690 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 snap rm mirror/image_12@snap 2023-10-11T11:09:44.557 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Removing snap: 100% complete...done. 2023-10-11T11:09:44.584 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_image_retry cluster2 mirror image_12 2023-10-11T11:09:44.584 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:09:44.584 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:09:44.585 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_12 2023-10-11T11:09:44.585 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0 1 2 4 8 16 32 2023-10-11T11:09:44.585 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0 2023-10-11T11:09:44.586 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_image cluster2 mirror image_12 2023-10-11T11:09:44.586 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:09:44.586 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:09:44.586 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_12 2023-10-11T11:09:44.586 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_12 2023-10-11T11:09:44.631 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 rm mirror/image_12 2023-10-11T11:09:44.749 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi111.stderr:2023-10-11T11:09:44.748+0000 7f1e68fcb700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x556984c761a0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-10-11T11:09:44.749 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi111.stderr:2023-10-11T11:09:44.748+0000 7f1e6f7d8700 -1 rbd::mirror::ImageReplayer: 0x5569846db680 [3/418ab5b5-2cf6-4b23-a645-1106d804c432] operator(): start failed: (85) Interrupted system call should be restarted 2023-10-11T11:09:46.564 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-10-11T11:09:46.567 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:09:46.568 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T11:09:46.568 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_13 2023-10-11T11:09:46.568 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_snapshot cluster2 mirror image_13 snap 2023-10-11T11:09:46.568 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:09:46.568 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:09:46.569 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_13 2023-10-11T11:09:46.569 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap=snap 2023-10-11T11:09:46.569 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 snap rm mirror/image_13@snap 2023-10-11T11:09:47.442 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Removing snap: 100% complete...done. 2023-10-11T11:09:47.463 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_image_retry cluster2 mirror image_13 2023-10-11T11:09:47.463 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:09:47.464 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:09:47.464 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_13 2023-10-11T11:09:47.464 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0 1 2 4 8 16 32 2023-10-11T11:09:47.465 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0 2023-10-11T11:09:47.465 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_image cluster2 mirror image_13 2023-10-11T11:09:47.466 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:09:47.466 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:09:47.466 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_13 2023-10-11T11:09:47.466 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_13 2023-10-11T11:09:47.512 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 rm mirror/image_13 2023-10-11T11:09:47.633 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi111.stderr:2023-10-11T11:09:47.632+0000 7f1e687ca700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x556984c77a00 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-10-11T11:09:47.633 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi111.stderr:2023-10-11T11:09:47.632+0000 7f1e6f7d8700 -1 rbd::mirror::ImageReplayer: 0x556984a07900 [3/bea67517-2558-4f8d-8c9b-71c5be4fe773] operator(): start failed: (85) Interrupted system call should be restarted 2023-10-11T11:09:49.582 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-10-11T11:09:49.585 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:09:49.586 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T11:09:49.586 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_14 2023-10-11T11:09:49.586 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_snapshot cluster2 mirror image_14 snap 2023-10-11T11:09:49.586 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:09:49.586 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:09:49.586 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_14 2023-10-11T11:09:49.587 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap=snap 2023-10-11T11:09:49.587 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 snap rm mirror/image_14@snap 2023-10-11T11:09:50.436 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Removing snap: 100% complete...done. 2023-10-11T11:09:50.465 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_image_retry cluster2 mirror image_14 2023-10-11T11:09:50.465 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:09:50.465 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:09:50.466 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_14 2023-10-11T11:09:50.466 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0 1 2 4 8 16 32 2023-10-11T11:09:50.466 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0 2023-10-11T11:09:50.467 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_image cluster2 mirror image_14 2023-10-11T11:09:50.467 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:09:50.467 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:09:50.467 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_14 2023-10-11T11:09:50.468 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_14 2023-10-11T11:09:50.513 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 rm mirror/image_14 2023-10-11T11:09:50.629 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi111.stderr:2023-10-11T11:09:50.628+0000 7f1e687ca700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x556984cc76c0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-10-11T11:09:50.630 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi111.stderr:2023-10-11T11:09:50.628+0000 7f1e6efd7700 -1 rbd::mirror::ImageReplayer: 0x556984a6c000 [3/cc29ef48-4a54-4049-9722-bc6c6b7caf39] operator(): start failed: (85) Interrupted system call should be restarted 2023-10-11T11:09:52.592 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-10-11T11:09:52.596 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:09:52.596 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T11:09:52.596 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_15 2023-10-11T11:09:52.596 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_snapshot cluster2 mirror image_15 snap 2023-10-11T11:09:52.597 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:09:52.597 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:09:52.597 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_15 2023-10-11T11:09:52.597 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap=snap 2023-10-11T11:09:52.597 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 snap rm mirror/image_15@snap 2023-10-11T11:09:53.472 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Removing snap: 100% complete...done. 2023-10-11T11:09:53.495 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_image_retry cluster2 mirror image_15 2023-10-11T11:09:53.495 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:09:53.496 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:09:53.496 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_15 2023-10-11T11:09:53.496 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0 1 2 4 8 16 32 2023-10-11T11:09:53.496 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0 2023-10-11T11:09:53.497 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_image cluster2 mirror image_15 2023-10-11T11:09:53.497 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:09:53.497 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:09:53.498 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_15 2023-10-11T11:09:53.498 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_15 2023-10-11T11:09:53.544 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 rm mirror/image_15 2023-10-11T11:09:53.659 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi111.stderr:2023-10-11T11:09:53.659+0000 7f1e687ca700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x556984cc7860 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-10-11T11:09:53.660 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi111.stderr:2023-10-11T11:09:53.659+0000 7f1e6efd7700 -1 rbd::mirror::ImageReplayer: 0x5569846e5680 [3/7f696646-7f90-4d09-a66b-990e8fbb123e] operator(): start failed: (85) Interrupted system call should be restarted 2023-10-11T11:09:55.565 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-10-11T11:09:55.569 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:09:55.569 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T11:09:55.569 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_16 2023-10-11T11:09:55.570 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_snapshot cluster2 mirror image_16 snap 2023-10-11T11:09:55.570 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:09:55.570 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:09:55.570 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_16 2023-10-11T11:09:55.570 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap=snap 2023-10-11T11:09:55.571 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 snap rm mirror/image_16@snap 2023-10-11T11:09:56.476 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Removing snap: 100% complete...done. 2023-10-11T11:09:56.497 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_image_retry cluster2 mirror image_16 2023-10-11T11:09:56.497 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:09:56.497 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:09:56.498 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_16 2023-10-11T11:09:56.498 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0 1 2 4 8 16 32 2023-10-11T11:09:56.498 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0 2023-10-11T11:09:56.499 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_image cluster2 mirror image_16 2023-10-11T11:09:56.500 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:09:56.500 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:09:56.500 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_16 2023-10-11T11:09:56.501 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_16 2023-10-11T11:09:56.542 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 rm mirror/image_16 2023-10-11T11:09:56.660 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi111.stderr:2023-10-11T11:09:56.659+0000 7f1e687ca700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x556984c76b60 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-10-11T11:09:56.660 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi111.stderr:2023-10-11T11:09:56.659+0000 7f1e6f7d8700 -1 rbd::mirror::ImageReplayer: 0x5569846daf00 [3/2ba7e4f0-06a3-4db0-aeee-64ab913d9bf0] operator(): start failed: (85) Interrupted system call should be restarted 2023-10-11T11:09:58.610 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-10-11T11:09:58.614 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:09:58.614 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T11:09:58.614 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_17 2023-10-11T11:09:58.614 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_snapshot cluster2 mirror image_17 snap 2023-10-11T11:09:58.614 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:09:58.615 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:09:58.615 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_17 2023-10-11T11:09:58.615 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap=snap 2023-10-11T11:09:58.615 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 snap rm mirror/image_17@snap 2023-10-11T11:09:59.485 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Removing snap: 100% complete...done. 2023-10-11T11:09:59.507 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_image_retry cluster2 mirror image_17 2023-10-11T11:09:59.508 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:09:59.508 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:09:59.508 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_17 2023-10-11T11:09:59.509 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0 1 2 4 8 16 32 2023-10-11T11:09:59.509 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0 2023-10-11T11:09:59.510 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_image cluster2 mirror image_17 2023-10-11T11:09:59.510 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:09:59.511 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:09:59.511 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_17 2023-10-11T11:09:59.511 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_17 2023-10-11T11:09:59.555 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 rm mirror/image_17 2023-10-11T11:09:59.677 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi111.stderr:2023-10-11T11:09:59.675+0000 7f1e68fcb700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x556984cc7a00 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-10-11T11:09:59.677 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi111.stderr:2023-10-11T11:09:59.675+0000 7f1e6efd7700 -1 rbd::mirror::ImageReplayer: 0x556984a6d180 [3/f3a165bf-82fd-4110-9215-0cf728d304ac] operator(): start failed: (85) Interrupted system call should be restarted 2023-10-11T11:09:59.714 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi111.stderr:2023-10-11T11:09:59.713+0000 7f1e687ca700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x556984cc76c0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-10-11T11:09:59.714 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi111.stderr:2023-10-11T11:09:59.713+0000 7f1e6f7d8700 -1 rbd::mirror::ImageReplayer: 0x556984a6d180 [3/f3a165bf-82fd-4110-9215-0cf728d304ac] operator(): start failed: (85) Interrupted system call should be restarted 2023-10-11T11:10:01.056 INFO:tasks.ceph.cluster1.osd.0.smithi050.stderr:2023-10-11T11:10:01.053+0000 7f61c5b4d700 -1 reset not still connected to 0x55651dbe00d0 2023-10-11T11:10:01.056 INFO:tasks.ceph.cluster1.osd.0.smithi050.stderr:2023-10-11T11:10:01.053+0000 7f61c5b4d700 -1 reset not still connected to 0x55651dbe04e0 2023-10-11T11:10:01.056 INFO:tasks.ceph.cluster1.osd.0.smithi050.stderr:2023-10-11T11:10:01.053+0000 7f61c5b4d700 -1 reset not still connected to 0x55651dbe0750 2023-10-11T11:10:01.057 INFO:tasks.ceph.cluster1.osd.0.smithi050.stderr:2023-10-11T11:10:01.053+0000 7f61c5b4d700 -1 reset not still connected to 0x55651de5eb60 2023-10-11T11:10:01.057 INFO:tasks.ceph.cluster1.osd.0.smithi050.stderr:2023-10-11T11:10:01.053+0000 7f61c5b4d700 -1 reset not still connected to 0x55651de5ed00 2023-10-11T11:10:01.057 INFO:tasks.ceph.cluster1.osd.0.smithi050.stderr:2023-10-11T11:10:01.054+0000 7f61c5b4d700 -1 reset not still connected to 0x55651e5fe270 2023-10-11T11:10:01.057 INFO:tasks.ceph.cluster1.osd.0.smithi050.stderr:2023-10-11T11:10:01.054+0000 7f61c5b4d700 -1 reset not still connected to 0x55651e5fe410 2023-10-11T11:10:01.058 INFO:tasks.ceph.cluster1.osd.0.smithi050.stderr:2023-10-11T11:10:01.054+0000 7f61c5b4d700 -1 reset not still connected to 0x55651e5fe820 2023-10-11T11:10:01.058 INFO:tasks.ceph.cluster1.osd.0.smithi050.stderr:2023-10-11T11:10:01.054+0000 7f61c5b4d700 -1 reset not still connected to 0x55651e5fe8f0 2023-10-11T11:10:01.058 INFO:tasks.ceph.cluster1.osd.0.smithi050.stderr:2023-10-11T11:10:01.054+0000 7f61c5b4d700 -1 reset not still connected to 0x55651e5fed00 2023-10-11T11:10:01.058 INFO:tasks.ceph.cluster1.osd.0.smithi050.stderr:2023-10-11T11:10:01.054+0000 7f61c5b4d700 -1 reset not still connected to 0x55651e5feea0 2023-10-11T11:10:01.059 INFO:tasks.ceph.cluster1.osd.0.smithi050.stderr:2023-10-11T11:10:01.054+0000 7f61c5b4d700 -1 reset not still connected to 0x55651e5ff110 2023-10-11T11:10:01.059 INFO:tasks.ceph.cluster1.osd.0.smithi050.stderr:2023-10-11T11:10:01.054+0000 7f61c5b4d700 -1 reset not still connected to 0x55651e5ff380 2023-10-11T11:10:01.059 INFO:tasks.ceph.cluster1.osd.0.smithi050.stderr:2023-10-11T11:10:01.054+0000 7f61c5b4d700 -1 reset not still connected to 0x55651e5ff520 2023-10-11T11:10:01.059 INFO:tasks.ceph.cluster1.osd.0.smithi050.stderr:2023-10-11T11:10:01.054+0000 7f61c5b4d700 -1 reset not still connected to 0x55651e5ff790 2023-10-11T11:10:01.060 INFO:tasks.ceph.cluster1.osd.0.smithi050.stderr:2023-10-11T11:10:01.054+0000 7f61c5b4d700 -1 reset not still connected to 0x55651ecf1ee0 2023-10-11T11:10:01.573 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-10-11T11:10:01.577 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:10:01.577 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T11:10:01.578 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_18 2023-10-11T11:10:01.578 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_snapshot cluster2 mirror image_18 snap 2023-10-11T11:10:01.578 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:10:01.578 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:10:01.579 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_18 2023-10-11T11:10:01.579 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap=snap 2023-10-11T11:10:01.579 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 snap rm mirror/image_18@snap 2023-10-11T11:10:02.503 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Removing snap: 100% complete...done. 2023-10-11T11:10:02.531 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_image_retry cluster2 mirror image_18 2023-10-11T11:10:02.531 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:10:02.531 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:10:02.532 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_18 2023-10-11T11:10:02.532 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0 1 2 4 8 16 32 2023-10-11T11:10:02.532 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0 2023-10-11T11:10:02.533 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_image cluster2 mirror image_18 2023-10-11T11:10:02.533 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:10:02.533 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:10:02.533 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_18 2023-10-11T11:10:02.533 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_18 2023-10-11T11:10:02.580 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 rm mirror/image_18 2023-10-11T11:10:02.696 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi111.stderr:2023-10-11T11:10:02.696+0000 7f1e68fcb700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x556984c77520 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-10-11T11:10:02.697 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi111.stderr:2023-10-11T11:10:02.696+0000 7f1e6f7d8700 -1 rbd::mirror::ImageReplayer: 0x556983bd1b80 [3/1526990d-f216-41c2-accc-30ee745ab7bd] operator(): start failed: (85) Interrupted system call should be restarted 2023-10-11T11:10:04.648 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-10-11T11:10:04.651 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:10:04.652 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T11:10:04.652 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_19 2023-10-11T11:10:04.652 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_snapshot cluster2 mirror image_19 snap 2023-10-11T11:10:04.652 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:10:04.653 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:10:04.653 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_19 2023-10-11T11:10:04.653 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap=snap 2023-10-11T11:10:04.654 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 snap rm mirror/image_19@snap 2023-10-11T11:10:05.448 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Removing snap: 100% complete...done. 2023-10-11T11:10:05.471 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_image_retry cluster2 mirror image_19 2023-10-11T11:10:05.471 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:10:05.471 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:10:05.472 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_19 2023-10-11T11:10:05.472 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0 1 2 4 8 16 32 2023-10-11T11:10:05.472 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0 2023-10-11T11:10:05.473 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_image cluster2 mirror image_19 2023-10-11T11:10:05.473 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:10:05.474 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:10:05.474 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_19 2023-10-11T11:10:05.474 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_19 2023-10-11T11:10:05.517 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 rm mirror/image_19 2023-10-11T11:10:05.630 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi111.stderr:2023-10-11T11:10:05.629+0000 7f1e687ca700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x556984cc7380 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-10-11T11:10:05.631 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi111.stderr:2023-10-11T11:10:05.629+0000 7f1e6efd7700 -1 rbd::mirror::ImageReplayer: 0x556983bd0a00 [3/0f059703-71ab-45c9-bae3-1fd34ca31402] operator(): start failed: (85) Interrupted system call should be restarted 2023-10-11T11:10:07.647 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-10-11T11:10:07.651 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:10:07.651 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T11:10:07.651 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_20 2023-10-11T11:10:07.651 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_snapshot cluster2 mirror image_20 snap 2023-10-11T11:10:07.652 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:10:07.652 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:10:07.652 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_20 2023-10-11T11:10:07.652 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap=snap 2023-10-11T11:10:07.652 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 snap rm mirror/image_20@snap 2023-10-11T11:10:08.526 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Removing snap: 100% complete...done. 2023-10-11T11:10:08.546 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_image_retry cluster2 mirror image_20 2023-10-11T11:10:08.546 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:10:08.546 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:10:08.547 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_20 2023-10-11T11:10:08.547 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0 1 2 4 8 16 32 2023-10-11T11:10:08.547 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0 2023-10-11T11:10:08.548 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_image cluster2 mirror image_20 2023-10-11T11:10:08.548 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:10:08.548 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:10:08.548 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_20 2023-10-11T11:10:08.548 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_20 2023-10-11T11:10:08.596 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 rm mirror/image_20 2023-10-11T11:10:08.716 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi111.stderr:2023-10-11T11:10:08.715+0000 7f1e68fcb700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5569849f8d00 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-10-11T11:10:08.716 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi111.stderr:2023-10-11T11:10:08.715+0000 7f1e6f7d8700 -1 rbd::mirror::ImageReplayer: 0x5569846db900 [3/46b9e65d-4379-44aa-a3c7-83f3b477d8e6] operator(): start failed: (85) Interrupted system call should be restarted 2023-10-11T11:10:10.574 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-10-11T11:10:10.578 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:10:10.578 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T11:10:10.578 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_21 2023-10-11T11:10:10.578 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_snapshot cluster2 mirror image_21 snap 2023-10-11T11:10:10.579 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:10:10.579 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:10:10.579 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_21 2023-10-11T11:10:10.579 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap=snap 2023-10-11T11:10:10.579 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 snap rm mirror/image_21@snap 2023-10-11T11:10:11.529 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Removing snap: 100% complete...done. 2023-10-11T11:10:11.549 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_image_retry cluster2 mirror image_21 2023-10-11T11:10:11.549 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:10:11.550 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:10:11.550 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_21 2023-10-11T11:10:11.550 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0 1 2 4 8 16 32 2023-10-11T11:10:11.550 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0 2023-10-11T11:10:11.551 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_image cluster2 mirror image_21 2023-10-11T11:10:11.551 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:10:11.551 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:10:11.552 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_21 2023-10-11T11:10:11.552 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_21 2023-10-11T11:10:11.596 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 rm mirror/image_21 2023-10-11T11:10:11.731 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi111.stderr:2023-10-11T11:10:11.730+0000 7f1e687ca700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5569849f81a0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-10-11T11:10:11.731 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi111.stderr:2023-10-11T11:10:11.730+0000 7f1e6f7d8700 -1 rbd::mirror::ImageReplayer: 0x556983bd1680 [3/11e7aef5-a626-4f47-9643-5ab9888eea8f] operator(): start failed: (85) Interrupted system call should be restarted 2023-10-11T11:10:13.683 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-10-11T11:10:13.687 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:10:13.687 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T11:10:13.687 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_22 2023-10-11T11:10:13.688 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_snapshot cluster2 mirror image_22 snap 2023-10-11T11:10:13.688 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:10:13.688 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:10:13.688 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_22 2023-10-11T11:10:13.688 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap=snap 2023-10-11T11:10:13.689 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 snap rm mirror/image_22@snap 2023-10-11T11:10:14.557 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Removing snap: 100% complete...done. 2023-10-11T11:10:14.581 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_image_retry cluster2 mirror image_22 2023-10-11T11:10:14.581 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:10:14.582 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:10:14.582 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_22 2023-10-11T11:10:14.582 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0 1 2 4 8 16 32 2023-10-11T11:10:14.582 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0 2023-10-11T11:10:14.583 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_image cluster2 mirror image_22 2023-10-11T11:10:14.584 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:10:14.584 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:10:14.584 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_22 2023-10-11T11:10:14.584 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_22 2023-10-11T11:10:14.628 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 rm mirror/image_22 2023-10-11T11:10:14.749 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi111.stderr:2023-10-11T11:10:14.747+0000 7f1e68fcb700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x556984cc7860 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-10-11T11:10:14.750 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi111.stderr:2023-10-11T11:10:14.747+0000 7f1e6efd7700 -1 rbd::mirror::ImageReplayer: 0x556983bd1900 [3/1288fb83-4aad-4242-bdd2-1944a6794332] operator(): start failed: (85) Interrupted system call should be restarted 2023-10-11T11:10:15.750 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.31802598717304875) > weight (0.25) 2023-10-11T11:10:15.750 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.5693281999418585) > weight (0.25) 2023-10-11T11:10:15.751 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.4137516315251676) > weight (0.25) 2023-10-11T11:10:15.751 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.3892641635669829) > weight (0.25) 2023-10-11T11:10:15.751 INFO:tasks.rbd_mirror_thrash:waiting for 77 secs before thrashing 2023-10-11T11:10:16.577 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-10-11T11:10:16.580 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:10:16.580 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T11:10:16.581 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_23 2023-10-11T11:10:16.581 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_snapshot cluster2 mirror image_23 snap 2023-10-11T11:10:16.581 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:10:16.581 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:10:16.581 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_23 2023-10-11T11:10:16.581 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap=snap 2023-10-11T11:10:16.582 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 snap rm mirror/image_23@snap 2023-10-11T11:10:17.453 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Removing snap: 100% complete...done. 2023-10-11T11:10:17.477 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_image_retry cluster2 mirror image_23 2023-10-11T11:10:17.477 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:10:17.477 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:10:17.478 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_23 2023-10-11T11:10:17.478 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0 1 2 4 8 16 32 2023-10-11T11:10:17.478 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0 2023-10-11T11:10:17.479 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_image cluster2 mirror image_23 2023-10-11T11:10:17.479 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:10:17.479 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:10:17.479 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_23 2023-10-11T11:10:17.479 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_23 2023-10-11T11:10:17.525 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 rm mirror/image_23 2023-10-11T11:10:18.255 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi111.stderr:2023-10-11T11:10:18.254+0000 7f1e687ca700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x556984cc6680 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-10-11T11:10:18.256 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi111.stderr:2023-10-11T11:10:18.254+0000 7f1e6efd7700 -1 rbd::mirror::ImageReplayer: 0x5569846e4280 [3/b3738f1e-cf2d-40b8-80a2-69299f4bee7b] operator(): start failed: (85) Interrupted system call should be restarted 2023-10-11T11:10:18.754 INFO:tasks.ceph.cluster1.osd.0.smithi050.stderr:2023-10-11T11:10:18.753+0000 7f61c5b4d700 -1 reset not still connected to 0x55651ce38340 2023-10-11T11:10:18.755 INFO:tasks.ceph.cluster1.osd.0.smithi050.stderr:2023-10-11T11:10:18.753+0000 7f61c5b4d700 -1 reset not still connected to 0x55651d2ca1a0 2023-10-11T11:10:18.755 INFO:tasks.ceph.cluster1.osd.0.smithi050.stderr:2023-10-11T11:10:18.753+0000 7f61c5b4d700 -1 reset not still connected to 0x55651d2ca340 2023-10-11T11:10:18.755 INFO:tasks.ceph.cluster1.osd.0.smithi050.stderr:2023-10-11T11:10:18.753+0000 7f61c5b4d700 -1 reset not still connected to 0x55651dbe00d0 2023-10-11T11:10:18.756 INFO:tasks.ceph.cluster1.osd.0.smithi050.stderr:2023-10-11T11:10:18.754+0000 7f61c5b4d700 -1 reset not still connected to 0x55651dbe04e0 2023-10-11T11:10:18.759 INFO:tasks.ceph.cluster1.osd.0.smithi050.stderr:2023-10-11T11:10:18.755+0000 7f61c5b4d700 -1 reset not still connected to 0x55651dbe0750 2023-10-11T11:10:18.760 INFO:tasks.ceph.cluster1.osd.0.smithi050.stderr:2023-10-11T11:10:18.755+0000 7f61c5b4d700 -1 reset not still connected to 0x55651dbe11e0 2023-10-11T11:10:18.760 INFO:tasks.ceph.cluster1.osd.0.smithi050.stderr:2023-10-11T11:10:18.755+0000 7f61c5b4d700 -1 reset not still connected to 0x55651e5fe270 2023-10-11T11:10:18.760 INFO:tasks.ceph.cluster1.osd.0.smithi050.stderr:2023-10-11T11:10:18.755+0000 7f61c5b4d700 -1 reset not still connected to 0x55651e5fe410 2023-10-11T11:10:18.760 INFO:tasks.ceph.cluster1.osd.0.smithi050.stderr:2023-10-11T11:10:18.755+0000 7f61c5b4d700 -1 reset not still connected to 0x55651e5fe820 2023-10-11T11:10:18.760 INFO:tasks.ceph.cluster1.osd.0.smithi050.stderr:2023-10-11T11:10:18.755+0000 7f61c5b4d700 -1 reset not still connected to 0x55651e5fe8f0 2023-10-11T11:10:18.761 INFO:tasks.ceph.cluster1.osd.0.smithi050.stderr:2023-10-11T11:10:18.755+0000 7f61c5b4d700 -1 reset not still connected to 0x55651e5fed00 2023-10-11T11:10:18.761 INFO:tasks.ceph.cluster1.osd.0.smithi050.stderr:2023-10-11T11:10:18.755+0000 7f61c5b4d700 -1 reset not still connected to 0x55651e5ff110 2023-10-11T11:10:18.761 INFO:tasks.ceph.cluster1.osd.0.smithi050.stderr:2023-10-11T11:10:18.757+0000 7f61c5b4d700 -1 reset not still connected to 0x55651e5ff380 2023-10-11T11:10:18.761 INFO:tasks.ceph.cluster1.osd.0.smithi050.stderr:2023-10-11T11:10:18.757+0000 7f61c5b4d700 -1 reset not still connected to 0x55651e5ff520 2023-10-11T11:10:18.761 INFO:tasks.ceph.cluster1.osd.0.smithi050.stderr:2023-10-11T11:10:18.757+0000 7f61c5b4d700 -1 reset not still connected to 0x55651e5ff790 2023-10-11T11:10:18.762 INFO:tasks.ceph.cluster1.osd.0.smithi050.stderr:2023-10-11T11:10:18.757+0000 7f61c5b4d700 -1 reset not still connected to 0x55651e9a51e0 2023-10-11T11:10:18.762 INFO:tasks.ceph.cluster1.osd.0.smithi050.stderr:2023-10-11T11:10:18.757+0000 7f61c5b4d700 -1 reset not still connected to 0x55651ecf1ee0 2023-10-11T11:10:19.585 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-10-11T11:10:19.589 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:10:19.589 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T11:10:19.589 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_24 2023-10-11T11:10:19.589 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_snapshot cluster2 mirror image_24 snap 2023-10-11T11:10:19.590 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:10:19.590 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:10:19.590 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_24 2023-10-11T11:10:19.590 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap=snap 2023-10-11T11:10:19.590 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 snap rm mirror/image_24@snap 2023-10-11T11:10:20.445 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Removing snap: 100% complete...done. 2023-10-11T11:10:20.470 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_image_retry cluster2 mirror image_24 2023-10-11T11:10:20.470 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:10:20.470 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:10:20.471 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_24 2023-10-11T11:10:20.471 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0 1 2 4 8 16 32 2023-10-11T11:10:20.471 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0 2023-10-11T11:10:20.472 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_image cluster2 mirror image_24 2023-10-11T11:10:20.472 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:10:20.472 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:10:20.472 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_24 2023-10-11T11:10:20.473 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_24 2023-10-11T11:10:20.515 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 rm mirror/image_24 2023-10-11T11:10:20.627 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi111.stderr:2023-10-11T11:10:20.626+0000 7f1e687ca700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5569849f8820 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-10-11T11:10:20.628 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi111.stderr:2023-10-11T11:10:20.626+0000 7f1e6f7d8700 -1 rbd::mirror::ImageReplayer: 0x5569846da000 [3/1d287991-2520-4320-9427-8214deba681d] operator(): start failed: (85) Interrupted system call should be restarted 2023-10-11T11:10:22.596 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-10-11T11:10:22.600 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:10:22.600 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T11:10:22.600 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_25 2023-10-11T11:10:22.601 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_snapshot cluster2 mirror image_25 snap 2023-10-11T11:10:22.601 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:10:22.601 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:10:22.601 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_25 2023-10-11T11:10:22.601 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap=snap 2023-10-11T11:10:22.601 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 snap rm mirror/image_25@snap 2023-10-11T11:10:23.470 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Removing snap: 100% complete...done. 2023-10-11T11:10:23.490 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_image_retry cluster2 mirror image_25 2023-10-11T11:10:23.490 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:10:23.491 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:10:23.491 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_25 2023-10-11T11:10:23.491 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0 1 2 4 8 16 32 2023-10-11T11:10:23.491 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0 2023-10-11T11:10:23.492 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_image cluster2 mirror image_25 2023-10-11T11:10:23.492 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:10:23.492 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:10:23.492 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_25 2023-10-11T11:10:23.493 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_25 2023-10-11T11:10:23.537 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 rm mirror/image_25 2023-10-11T11:10:23.654 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi111.stderr:2023-10-11T11:10:23.653+0000 7f1e687ca700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5569849f89c0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-10-11T11:10:23.654 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi111.stderr:2023-10-11T11:10:23.653+0000 7f1e6f7d8700 -1 rbd::mirror::ImageReplayer: 0x5569846dbb80 [3/48104021-9df0-46cf-a4dc-ab4fb67e4fe1] operator(): start failed: (85) Interrupted system call should be restarted 2023-10-11T11:10:25.569 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-10-11T11:10:25.573 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:10:25.573 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T11:10:25.573 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_26 2023-10-11T11:10:25.573 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_snapshot cluster2 mirror image_26 snap 2023-10-11T11:10:25.574 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:10:25.574 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:10:25.574 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_26 2023-10-11T11:10:25.574 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap=snap 2023-10-11T11:10:25.574 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 snap rm mirror/image_26@snap 2023-10-11T11:10:26.484 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Removing snap: 100% complete...done. 2023-10-11T11:10:26.502 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_image_retry cluster2 mirror image_26 2023-10-11T11:10:26.503 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:10:26.503 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:10:26.503 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_26 2023-10-11T11:10:26.503 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0 1 2 4 8 16 32 2023-10-11T11:10:26.503 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0 2023-10-11T11:10:26.504 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_image cluster2 mirror image_26 2023-10-11T11:10:26.504 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:10:26.505 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:10:26.505 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_26 2023-10-11T11:10:26.505 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_26 2023-10-11T11:10:26.550 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 rm mirror/image_26 2023-10-11T11:10:26.671 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi111.stderr:2023-10-11T11:10:26.670+0000 7f1e68fcb700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x556984cc6340 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-10-11T11:10:26.671 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi111.stderr:2023-10-11T11:10:26.670+0000 7f1e6efd7700 -1 rbd::mirror::ImageReplayer: 0x5569846e4c80 [3/76bafcc4-4e59-4baf-875a-22631193e31e] operator(): start failed: (85) Interrupted system call should be restarted 2023-10-11T11:10:27.799 INFO:tasks.ceph.cluster1.osd.0.smithi050.stderr:2023-10-11T11:10:27.796+0000 7f61c5b4d700 -1 reset not still connected to 0x55651dbe04e0 2023-10-11T11:10:27.800 INFO:tasks.ceph.cluster1.osd.0.smithi050.stderr:2023-10-11T11:10:27.796+0000 7f61c5b4d700 -1 reset not still connected to 0x55651dbe0750 2023-10-11T11:10:27.800 INFO:tasks.ceph.cluster1.osd.0.smithi050.stderr:2023-10-11T11:10:27.796+0000 7f61c5b4d700 -1 reset not still connected to 0x55651e5fe270 2023-10-11T11:10:27.800 INFO:tasks.ceph.cluster1.osd.0.smithi050.stderr:2023-10-11T11:10:27.796+0000 7f61c5b4d700 -1 reset not still connected to 0x55651e5fe410 2023-10-11T11:10:27.800 INFO:tasks.ceph.cluster1.osd.0.smithi050.stderr:2023-10-11T11:10:27.796+0000 7f61c5b4d700 -1 reset not still connected to 0x55651e5fe820 2023-10-11T11:10:27.800 INFO:tasks.ceph.cluster1.osd.0.smithi050.stderr:2023-10-11T11:10:27.796+0000 7f61c5b4d700 -1 reset not still connected to 0x55651e5fe8f0 2023-10-11T11:10:27.801 INFO:tasks.ceph.cluster1.osd.0.smithi050.stderr:2023-10-11T11:10:27.796+0000 7f61c5b4d700 -1 reset not still connected to 0x55651e5fed00 2023-10-11T11:10:27.801 INFO:tasks.ceph.cluster1.osd.0.smithi050.stderr:2023-10-11T11:10:27.796+0000 7f61c5b4d700 -1 reset not still connected to 0x55651e5ff380 2023-10-11T11:10:27.801 INFO:tasks.ceph.cluster1.osd.0.smithi050.stderr:2023-10-11T11:10:27.796+0000 7f61c5b4d700 -1 reset not still connected to 0x55651e5ff520 2023-10-11T11:10:27.801 INFO:tasks.ceph.cluster1.osd.0.smithi050.stderr:2023-10-11T11:10:27.796+0000 7f61c5b4d700 -1 reset not still connected to 0x55651e5ff790 2023-10-11T11:10:27.801 INFO:tasks.ceph.cluster1.osd.0.smithi050.stderr:2023-10-11T11:10:27.797+0000 7f61c5b4d700 -1 reset not still connected to 0x55651ecf1ee0 2023-10-11T11:10:28.630 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-10-11T11:10:28.634 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:10:28.634 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T11:10:28.635 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_27 2023-10-11T11:10:28.635 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_snapshot cluster2 mirror image_27 snap 2023-10-11T11:10:28.635 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:10:28.635 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:10:28.635 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_27 2023-10-11T11:10:28.636 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap=snap 2023-10-11T11:10:28.636 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 snap rm mirror/image_27@snap 2023-10-11T11:10:29.492 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Removing snap: 100% complete...done. 2023-10-11T11:10:29.515 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_image_retry cluster2 mirror image_27 2023-10-11T11:10:29.515 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:10:29.515 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:10:29.516 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_27 2023-10-11T11:10:29.516 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0 1 2 4 8 16 32 2023-10-11T11:10:29.516 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0 2023-10-11T11:10:29.517 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_image cluster2 mirror image_27 2023-10-11T11:10:29.517 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:10:29.517 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:10:29.518 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_27 2023-10-11T11:10:29.518 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_27 2023-10-11T11:10:29.565 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 rm mirror/image_27 2023-10-11T11:10:29.682 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi111.stderr:2023-10-11T11:10:29.681+0000 7f1e687ca700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5569849f9040 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-10-11T11:10:29.682 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi111.stderr:2023-10-11T11:10:29.681+0000 7f1e6f7d8700 -1 rbd::mirror::ImageReplayer: 0x556984a6d680 [3/fed85990-e790-4702-a20f-70d066da94cd] operator(): start failed: (85) Interrupted system call should be restarted 2023-10-11T11:10:29.712 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi111.stderr:2023-10-11T11:10:29.711+0000 7f1e68fcb700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x556983eba4e0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-10-11T11:10:29.712 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi111.stderr:2023-10-11T11:10:29.711+0000 7f1e6f7d8700 -1 rbd::mirror::ImageReplayer: 0x556984a6d680 [3/fed85990-e790-4702-a20f-70d066da94cd] operator(): start failed: (85) Interrupted system call should be restarted 2023-10-11T11:10:31.575 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-10-11T11:10:31.579 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:10:31.580 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T11:10:31.580 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_28 2023-10-11T11:10:31.580 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_snapshot cluster2 mirror image_28 snap 2023-10-11T11:10:31.580 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:10:31.580 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:10:31.581 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_28 2023-10-11T11:10:31.581 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap=snap 2023-10-11T11:10:31.581 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 snap rm mirror/image_28@snap 2023-10-11T11:10:32.511 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Removing snap: 100% complete...done. 2023-10-11T11:10:32.537 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_image_retry cluster2 mirror image_28 2023-10-11T11:10:32.537 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:10:32.537 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:10:32.537 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_28 2023-10-11T11:10:32.538 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0 1 2 4 8 16 32 2023-10-11T11:10:32.538 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0 2023-10-11T11:10:32.539 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_image cluster2 mirror image_28 2023-10-11T11:10:32.539 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:10:32.539 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:10:32.539 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_28 2023-10-11T11:10:32.539 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_28 2023-10-11T11:10:32.583 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 rm mirror/image_28 2023-10-11T11:10:32.705 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi111.stderr:2023-10-11T11:10:32.704+0000 7f1e687ca700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x556983ebd860 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-10-11T11:10:32.706 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi111.stderr:2023-10-11T11:10:32.704+0000 7f1e6efd7700 -1 rbd::mirror::ImageReplayer: 0x5569846e4000 [3/48b201ee-f8ea-4845-97ba-749a1f35906b] operator(): start failed: (85) Interrupted system call should be restarted 2023-10-11T11:10:34.702 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-10-11T11:10:34.705 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:10:34.705 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T11:10:34.706 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_29 2023-10-11T11:10:34.706 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_snapshot cluster2 mirror image_29 snap 2023-10-11T11:10:34.706 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:10:34.706 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:10:34.706 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_29 2023-10-11T11:10:34.707 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap=snap 2023-10-11T11:10:34.707 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 snap rm mirror/image_29@snap 2023-10-11T11:10:35.455 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Removing snap: 100% complete...done. 2023-10-11T11:10:35.468 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_image_retry cluster2 mirror image_29 2023-10-11T11:10:35.469 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:10:35.469 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:10:35.469 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_29 2023-10-11T11:10:35.469 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0 1 2 4 8 16 32 2023-10-11T11:10:35.470 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0 2023-10-11T11:10:35.470 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_image cluster2 mirror image_29 2023-10-11T11:10:35.471 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:10:35.471 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:10:35.471 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_29 2023-10-11T11:10:35.471 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_29 2023-10-11T11:10:35.516 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 rm mirror/image_29 2023-10-11T11:10:35.744 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi111.stderr:2023-10-11T11:10:35.743+0000 7f1e68fcb700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x556984a11a00 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-10-11T11:10:35.744 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi111.stderr:2023-10-11T11:10:35.743+0000 7f1e6f7d8700 -1 rbd::mirror::ImageReplayer: 0x556984a6c780 [3/d589d227-75bc-45bf-a707-46d8d0068324] operator(): start failed: (85) Interrupted system call should be restarted 2023-10-11T11:10:37.703 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-10-11T11:10:37.707 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:10:37.707 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T11:10:37.707 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_30 2023-10-11T11:10:37.707 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_snapshot cluster2 mirror image_30 snap 2023-10-11T11:10:37.708 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:10:37.708 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:10:37.708 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_30 2023-10-11T11:10:37.708 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap=snap 2023-10-11T11:10:37.708 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 snap rm mirror/image_30@snap 2023-10-11T11:10:38.570 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Removing snap: 100% complete...done. 2023-10-11T11:10:38.584 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_image_retry cluster2 mirror image_30 2023-10-11T11:10:38.584 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:10:38.584 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:10:38.585 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_30 2023-10-11T11:10:38.585 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0 1 2 4 8 16 32 2023-10-11T11:10:38.585 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0 2023-10-11T11:10:38.586 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_image cluster2 mirror image_30 2023-10-11T11:10:38.586 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:10:38.586 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:10:38.586 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_30 2023-10-11T11:10:38.586 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_30 2023-10-11T11:10:38.587 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.9085009595136704) > weight (0.25) 2023-10-11T11:10:38.587 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.41004189922707235) > weight (0.25) 2023-10-11T11:10:38.587 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.2 2023-10-11T11:10:38.588 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Sent signal 15 2023-10-11T11:10:38.588 INFO:tasks.rbd_mirror_thrash:waiting for 5 secs before reviving daemons 2023-10-11T11:10:38.588 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi111.stderr:2023-10-11T11:10:38.587+0000 7f2c09134700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 (PID: 121800) UID: 1000 2023-10-11T11:10:38.643 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 rm mirror/image_30 2023-10-11T11:10:38.758 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi111.stderr:2023-10-11T11:10:38.757+0000 7f1e687ca700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x556983ebda00 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-10-11T11:10:38.759 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi111.stderr:2023-10-11T11:10:38.758+0000 7f1e6efd7700 -1 rbd::mirror::ImageReplayer: 0x5569846db400 [3/3d66dfc7-26fa-4c9e-b7cc-ebb9eb40ba6a] operator(): start failed: (85) Interrupted system call should be restarted 2023-10-11T11:10:40.578 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-10-11T11:10:40.581 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:10:40.581 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T11:10:40.581 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_31 2023-10-11T11:10:40.582 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_snapshot cluster2 mirror image_31 snap 2023-10-11T11:10:40.582 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:10:40.582 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:10:40.582 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_31 2023-10-11T11:10:40.582 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap=snap 2023-10-11T11:10:40.583 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 snap rm mirror/image_31@snap 2023-10-11T11:10:41.574 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Removing snap: 100% complete...done. 2023-10-11T11:10:41.592 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_image_retry cluster2 mirror image_31 2023-10-11T11:10:41.592 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:10:41.592 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:10:41.593 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_31 2023-10-11T11:10:41.593 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0 1 2 4 8 16 32 2023-10-11T11:10:41.593 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0 2023-10-11T11:10:41.594 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_image cluster2 mirror image_31 2023-10-11T11:10:41.594 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:10:41.594 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:10:41.594 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_31 2023-10-11T11:10:41.595 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_31 2023-10-11T11:10:41.641 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 rm mirror/image_31 2023-10-11T11:10:41.782 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi111.stderr:2023-10-11T11:10:41.782+0000 7f1e687ca700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x556983eeba00 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-10-11T11:10:41.783 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi111.stderr:2023-10-11T11:10:41.782+0000 7f1e6efd7700 -1 rbd::mirror::ImageReplayer: 0x556984a6ca00 [3/da5623f9-4467-416e-af1d-0ee95e25c12e] operator(): start failed: (85) Interrupted system call should be restarted 2023-10-11T11:10:43.588 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.2 2023-10-11T11:10:43.589 INFO:teuthology.orchestra.run:waiting for 600 2023-10-11T11:10:43.589 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.2 2023-10-11T11:10:43.590 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Restarting daemon 2023-10-11T11:10:43.590 DEBUG:teuthology.orchestra.run.smithi111:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 2023-10-11T11:10:43.593 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Started 2023-10-11T11:10:43.593 INFO:tasks.rbd_mirror_thrash:waiting for 112 secs before thrashing 2023-10-11T11:10:43.725 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-10-11T11:10:43.728 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:10:43.728 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T11:10:43.729 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_32 2023-10-11T11:10:43.729 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_snapshot cluster2 mirror image_32 snap 2023-10-11T11:10:43.729 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:10:43.729 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:10:43.729 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_32 2023-10-11T11:10:43.730 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap=snap 2023-10-11T11:10:43.730 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 snap rm mirror/image_32@snap 2023-10-11T11:10:44.585 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Removing snap: 100% complete...done. 2023-10-11T11:10:44.609 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_image_retry cluster2 mirror image_32 2023-10-11T11:10:44.610 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:10:44.610 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:10:44.610 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_32 2023-10-11T11:10:44.611 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0 1 2 4 8 16 32 2023-10-11T11:10:44.611 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0 2023-10-11T11:10:44.611 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_image cluster2 mirror image_32 2023-10-11T11:10:44.612 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:10:44.612 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:10:44.612 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_32 2023-10-11T11:10:44.613 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_32 2023-10-11T11:10:44.654 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 rm mirror/image_32 2023-10-11T11:10:44.774 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi111.stderr:2023-10-11T11:10:44.773+0000 7f1e687ca700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x556984a10680 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-10-11T11:10:44.774 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi111.stderr:2023-10-11T11:10:44.773+0000 7f1e6f7d8700 -1 rbd::mirror::ImageReplayer: 0x556984a07680 [3/b4fc3ca8-51a4-45f4-a987-97af54ee49dc] operator(): start failed: (85) Interrupted system call should be restarted 2023-10-11T11:10:46.585 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-10-11T11:10:46.589 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:10:46.589 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T11:10:46.589 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_33 2023-10-11T11:10:46.589 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_snapshot cluster2 mirror image_33 snap 2023-10-11T11:10:46.590 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:10:46.590 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:10:46.590 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_33 2023-10-11T11:10:46.590 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap=snap 2023-10-11T11:10:46.590 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 snap rm mirror/image_33@snap 2023-10-11T11:10:47.461 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Removing snap: 100% complete...done. 2023-10-11T11:10:47.477 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_image_retry cluster2 mirror image_33 2023-10-11T11:10:47.477 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:10:47.478 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:10:47.478 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_33 2023-10-11T11:10:47.478 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0 1 2 4 8 16 32 2023-10-11T11:10:47.478 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0 2023-10-11T11:10:47.479 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_image cluster2 mirror image_33 2023-10-11T11:10:47.479 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:10:47.479 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:10:47.479 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_33 2023-10-11T11:10:47.480 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_33 2023-10-11T11:10:47.521 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 rm mirror/image_33 2023-10-11T11:10:47.647 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi111.stderr:2023-10-11T11:10:47.646+0000 7f1e68fcb700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x556984a10000 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-10-11T11:10:47.648 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi111.stderr:2023-10-11T11:10:47.647+0000 7f1e6f7d8700 -1 rbd::mirror::ImageReplayer: 0x556984a06f00 [3/a8456df7-6856-4049-9b99-fded34440c1f] operator(): start failed: (85) Interrupted system call should be restarted 2023-10-11T11:10:49.604 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-10-11T11:10:49.608 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:10:49.608 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T11:10:49.608 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_34 2023-10-11T11:10:49.608 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_snapshot cluster2 mirror image_34 snap 2023-10-11T11:10:49.609 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:10:49.609 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:10:49.609 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_34 2023-10-11T11:10:49.609 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap=snap 2023-10-11T11:10:49.609 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 snap rm mirror/image_34@snap 2023-10-11T11:10:50.457 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Removing snap: 100% complete...done. 2023-10-11T11:10:50.480 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_image_retry cluster2 mirror image_34 2023-10-11T11:10:50.480 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:10:50.480 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:10:50.480 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_34 2023-10-11T11:10:50.481 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0 1 2 4 8 16 32 2023-10-11T11:10:50.481 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0 2023-10-11T11:10:50.482 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_image cluster2 mirror image_34 2023-10-11T11:10:50.482 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:10:50.482 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:10:50.482 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_34 2023-10-11T11:10:50.483 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_34 2023-10-11T11:10:50.527 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 rm mirror/image_34 2023-10-11T11:10:50.848 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi111.stderr:2023-10-11T11:10:50.847+0000 7f1e68fcb700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x556984373a00 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-10-11T11:10:50.848 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi111.stderr:2023-10-11T11:10:50.847+0000 7f1e6efd7700 -1 rbd::mirror::ImageReplayer: 0x5569846e5180 [3/78a70dcb-6515-4835-bfd0-d59363f95aba] operator(): start failed: (85) Interrupted system call should be restarted 2023-10-11T11:10:52.610 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-10-11T11:10:52.614 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:10:52.614 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T11:10:52.614 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_35 2023-10-11T11:10:52.615 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_snapshot cluster2 mirror image_35 snap 2023-10-11T11:10:52.615 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:10:52.615 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:10:52.615 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_35 2023-10-11T11:10:52.615 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap=snap 2023-10-11T11:10:52.616 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 snap rm mirror/image_35@snap 2023-10-11T11:10:53.477 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Removing snap: 100% complete...done. 2023-10-11T11:10:53.503 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_image_retry cluster2 mirror image_35 2023-10-11T11:10:53.504 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:10:53.504 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:10:53.504 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_35 2023-10-11T11:10:53.505 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0 1 2 4 8 16 32 2023-10-11T11:10:53.505 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0 2023-10-11T11:10:53.506 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_image cluster2 mirror image_35 2023-10-11T11:10:53.506 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:10:53.506 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:10:53.507 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_35 2023-10-11T11:10:53.507 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_35 2023-10-11T11:10:53.549 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 rm mirror/image_35 2023-10-11T11:10:53.664 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi111.stderr:2023-10-11T11:10:53.663+0000 7f1e687ca700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5569843729c0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-10-11T11:10:53.664 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi111.stderr:2023-10-11T11:10:53.663+0000 7f1e6efd7700 -1 rbd::mirror::ImageReplayer: 0x556984a6c500 [3/d4ebf971-35de-4686-a8a6-e84c09701f07] operator(): start failed: (85) Interrupted system call should be restarted 2023-10-11T11:10:55.600 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-10-11T11:10:55.603 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:10:55.603 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T11:10:55.603 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_36 2023-10-11T11:10:55.603 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_snapshot cluster2 mirror image_36 snap 2023-10-11T11:10:55.604 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:10:55.604 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:10:55.604 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_36 2023-10-11T11:10:55.604 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap=snap 2023-10-11T11:10:55.604 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 snap rm mirror/image_36@snap 2023-10-11T11:10:56.499 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Removing snap: 100% complete...done. 2023-10-11T11:10:56.517 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_image_retry cluster2 mirror image_36 2023-10-11T11:10:56.517 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:10:56.517 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:10:56.517 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_36 2023-10-11T11:10:56.517 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0 1 2 4 8 16 32 2023-10-11T11:10:56.518 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0 2023-10-11T11:10:56.518 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_image cluster2 mirror image_36 2023-10-11T11:10:56.519 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:10:56.519 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:10:56.519 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_36 2023-10-11T11:10:56.519 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_36 2023-10-11T11:10:56.563 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 rm mirror/image_36 2023-10-11T11:10:56.683 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi111.stderr:2023-10-11T11:10:56.682+0000 7f1e68fcb700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x556984a111e0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-10-11T11:10:56.762 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi111.stderr:2023-10-11T11:10:56.682+0000 7f1e6f7d8700 -1 rbd::mirror::ImageReplayer: 0x5569846dac80 [3/26586d31-8ccc-4ee9-b1da-f7f6be6b7fa9] operator(): start failed: (85) Interrupted system call should be restarted 2023-10-11T11:10:58.640 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-10-11T11:10:58.643 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:10:58.644 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T11:10:58.644 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_37 2023-10-11T11:10:58.644 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_snapshot cluster2 mirror image_37 snap 2023-10-11T11:10:58.644 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:10:58.644 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:10:58.645 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_37 2023-10-11T11:10:58.645 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap=snap 2023-10-11T11:10:58.645 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 snap rm mirror/image_37@snap 2023-10-11T11:10:59.508 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Removing snap: 100% complete...done. 2023-10-11T11:10:59.522 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_image_retry cluster2 mirror image_37 2023-10-11T11:10:59.522 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:10:59.523 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:10:59.523 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_37 2023-10-11T11:10:59.523 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0 1 2 4 8 16 32 2023-10-11T11:10:59.523 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0 2023-10-11T11:10:59.524 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_image cluster2 mirror image_37 2023-10-11T11:10:59.524 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:10:59.524 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:10:59.524 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_37 2023-10-11T11:10:59.524 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_37 2023-10-11T11:10:59.567 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 rm mirror/image_37 2023-10-11T11:10:59.689 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi111.stderr:2023-10-11T11:10:59.688+0000 7f1e68fcb700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x556984373040 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-10-11T11:10:59.690 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi111.stderr:2023-10-11T11:10:59.688+0000 7f1e6efd7700 -1 rbd::mirror::ImageReplayer: 0x5569846da500 [3/71f091e9-defc-4548-9e77-73505b063ba4] operator(): start failed: (85) Interrupted system call should be restarted 2023-10-11T11:10:59.705 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi111.stderr:2023-10-11T11:10:59.704+0000 7f1e68fcb700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x556984640b60 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-10-11T11:10:59.705 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi111.stderr:2023-10-11T11:10:59.704+0000 7f1e6f7d8700 -1 rbd::mirror::ImageReplayer: 0x5569846da500 [3/71f091e9-defc-4548-9e77-73505b063ba4] operator(): start failed: (85) Interrupted system call should be restarted 2023-10-11T11:10:59.710 INFO:tasks.ceph.cluster2.osd.2.smithi111.stderr:2023-10-11T11:10:59.708+0000 7fa35bd9b700 -1 reset not still connected to 0x556400134820 2023-10-11T11:10:59.710 INFO:tasks.ceph.cluster2.osd.2.smithi111.stderr:2023-10-11T11:10:59.708+0000 7fa35bd9b700 -1 reset not still connected to 0x556400134a90 2023-10-11T11:10:59.713 INFO:tasks.ceph.cluster2.osd.2.smithi111.stderr:2023-10-11T11:10:59.712+0000 7fa35bd9b700 -1 reset not still connected to 0x55641c38dad0 2023-10-11T11:10:59.713 INFO:tasks.ceph.cluster2.osd.2.smithi111.stderr:2023-10-11T11:10:59.712+0000 7fa35bd9b700 -1 reset not still connected to 0x5564219388f0 2023-10-11T11:11:01.589 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-10-11T11:11:01.592 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:11:01.593 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T11:11:01.593 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_38 2023-10-11T11:11:01.593 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_snapshot cluster2 mirror image_38 snap 2023-10-11T11:11:01.593 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:11:01.594 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:11:01.594 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_38 2023-10-11T11:11:01.594 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap=snap 2023-10-11T11:11:01.594 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 snap rm mirror/image_38@snap 2023-10-11T11:11:02.514 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Removing snap: 100% complete...done. 2023-10-11T11:11:02.536 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_image_retry cluster2 mirror image_38 2023-10-11T11:11:02.537 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:11:02.537 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:11:02.537 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_38 2023-10-11T11:11:02.537 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0 1 2 4 8 16 32 2023-10-11T11:11:02.538 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0 2023-10-11T11:11:02.538 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_image cluster2 mirror image_38 2023-10-11T11:11:02.538 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:11:02.539 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:11:02.539 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_38 2023-10-11T11:11:02.539 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_38 2023-10-11T11:11:02.582 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 rm mirror/image_38 2023-10-11T11:11:02.702 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi111.stderr:2023-10-11T11:11:02.701+0000 7f1e687ca700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x556984a111e0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-10-11T11:11:02.702 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi111.stderr:2023-10-11T11:11:02.701+0000 7f1e6efd7700 -1 rbd::mirror::ImageReplayer: 0x5569846e5b80 [3/8f0a6f87-4fd6-4ab1-8d9b-69b67766f0e3] operator(): start failed: (85) Interrupted system call should be restarted 2023-10-11T11:11:04.657 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-10-11T11:11:04.660 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:11:04.660 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T11:11:04.661 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_39 2023-10-11T11:11:04.661 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_snapshot cluster2 mirror image_39 snap 2023-10-11T11:11:04.661 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:11:04.661 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:11:04.661 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_39 2023-10-11T11:11:04.662 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap=snap 2023-10-11T11:11:04.662 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 snap rm mirror/image_39@snap 2023-10-11T11:11:05.455 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Removing snap: 100% complete...done. 2023-10-11T11:11:05.478 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_image_retry cluster2 mirror image_39 2023-10-11T11:11:05.478 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:11:05.478 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:11:05.479 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_39 2023-10-11T11:11:05.479 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0 1 2 4 8 16 32 2023-10-11T11:11:05.479 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0 2023-10-11T11:11:05.480 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_image cluster2 mirror image_39 2023-10-11T11:11:05.480 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:11:05.480 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:11:05.480 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_39 2023-10-11T11:11:05.480 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_39 2023-10-11T11:11:05.524 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 rm mirror/image_39 2023-10-11T11:11:05.851 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi111.stderr:2023-10-11T11:11:05.851+0000 7f1e68fcb700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55698468bd40 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-10-11T11:11:05.852 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi111.stderr:2023-10-11T11:11:05.851+0000 7f1e6f7d8700 -1 rbd::mirror::ImageReplayer: 0x5569846daa00 [3/20fcc198-7ca6-4408-82dd-c95c583a202d] operator(): start failed: (85) Interrupted system call should be restarted 2023-10-11T11:11:07.693 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-10-11T11:11:07.697 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:11:07.697 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T11:11:07.697 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_40 2023-10-11T11:11:07.697 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_snapshot cluster2 mirror image_40 snap 2023-10-11T11:11:07.697 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:11:07.698 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:11:07.698 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_40 2023-10-11T11:11:07.698 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap=snap 2023-10-11T11:11:07.698 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 snap rm mirror/image_40@snap 2023-10-11T11:11:08.550 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Removing snap: 100% complete...done. 2023-10-11T11:11:08.567 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_image_retry cluster2 mirror image_40 2023-10-11T11:11:08.567 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:11:08.567 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:11:08.568 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_40 2023-10-11T11:11:08.568 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0 1 2 4 8 16 32 2023-10-11T11:11:08.568 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0 2023-10-11T11:11:08.569 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_image cluster2 mirror image_40 2023-10-11T11:11:08.569 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:11:08.569 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:11:08.569 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_40 2023-10-11T11:11:08.569 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_40 2023-10-11T11:11:08.613 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 rm mirror/image_40 2023-10-11T11:11:08.728 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi111.stderr:2023-10-11T11:11:08.727+0000 7f1e687ca700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x556984a10680 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-10-11T11:11:08.728 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi111.stderr:2023-10-11T11:11:08.727+0000 7f1e6efd7700 -1 rbd::mirror::ImageReplayer: 0x556984a06280 [3/932fa6e9-efac-4d1a-a4b3-2f155e3296d4] operator(): start failed: (85) Interrupted system call should be restarted 2023-10-11T11:11:09.779 INFO:tasks.ceph.cluster1.osd.0.smithi050.stderr:2023-10-11T11:11:09.778+0000 7f61c5b4d700 -1 reset not still connected to 0x55651ecf1ee0 2023-10-11T11:11:10.601 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-10-11T11:11:10.604 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:11:10.605 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T11:11:10.605 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_41 2023-10-11T11:11:10.605 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_snapshot cluster2 mirror image_41 snap 2023-10-11T11:11:10.605 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:11:10.605 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:11:10.606 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_41 2023-10-11T11:11:10.606 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap=snap 2023-10-11T11:11:10.606 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 snap rm mirror/image_41@snap 2023-10-11T11:11:11.549 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Removing snap: 100% complete...done. 2023-10-11T11:11:11.576 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_image_retry cluster2 mirror image_41 2023-10-11T11:11:11.577 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:11:11.577 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:11:11.577 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_41 2023-10-11T11:11:11.577 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0 1 2 4 8 16 32 2023-10-11T11:11:11.577 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0 2023-10-11T11:11:11.578 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_image cluster2 mirror image_41 2023-10-11T11:11:11.578 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:11:11.579 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:11:11.579 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_41 2023-10-11T11:11:11.579 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_41 2023-10-11T11:11:11.621 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 rm mirror/image_41 2023-10-11T11:11:11.751 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi111.stderr:2023-10-11T11:11:11.749+0000 7f1e68fcb700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x556984a116c0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-10-11T11:11:11.752 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi111.stderr:2023-10-11T11:11:11.749+0000 7f1e6efd7700 -1 rbd::mirror::ImageReplayer: 0x556984a06c80 [3/981fef91-675f-4aae-8d59-cba97f3e8943] operator(): start failed: (85) Interrupted system call should be restarted 2023-10-11T11:11:13.681 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-10-11T11:11:13.685 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:11:13.685 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T11:11:13.685 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_42 2023-10-11T11:11:13.685 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_snapshot cluster2 mirror image_42 snap 2023-10-11T11:11:13.685 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:11:13.686 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:11:13.686 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_42 2023-10-11T11:11:13.686 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap=snap 2023-10-11T11:11:13.686 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 snap rm mirror/image_42@snap 2023-10-11T11:11:14.562 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Removing snap: 100% complete...done. 2023-10-11T11:11:14.574 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_image_retry cluster2 mirror image_42 2023-10-11T11:11:14.574 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:11:14.575 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:11:14.575 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_42 2023-10-11T11:11:14.575 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0 1 2 4 8 16 32 2023-10-11T11:11:14.575 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0 2023-10-11T11:11:14.576 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_image cluster2 mirror image_42 2023-10-11T11:11:14.576 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:11:14.576 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:11:14.576 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_42 2023-10-11T11:11:14.577 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_42 2023-10-11T11:11:14.622 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 rm mirror/image_42 2023-10-11T11:11:14.745 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi111.stderr:2023-10-11T11:11:14.744+0000 7f1e687ca700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55698468b6c0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-10-11T11:11:14.746 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi111.stderr:2023-10-11T11:11:14.744+0000 7f1e6f7d8700 -1 rbd::mirror::ImageReplayer: 0x556984a06000 [3/8fbaa844-5c7f-4d9f-b667-ac0ce0c88d90] operator(): start failed: (85) Interrupted system call should be restarted 2023-10-11T11:11:16.597 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-10-11T11:11:16.600 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:11:16.601 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T11:11:16.601 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_43 2023-10-11T11:11:16.601 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_snapshot cluster2 mirror image_43 snap 2023-10-11T11:11:16.601 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:11:16.601 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:11:16.602 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_43 2023-10-11T11:11:16.602 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap=snap 2023-10-11T11:11:16.602 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 snap rm mirror/image_43@snap 2023-10-11T11:11:17.469 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Removing snap: 100% complete...done. 2023-10-11T11:11:17.489 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_image_retry cluster2 mirror image_43 2023-10-11T11:11:17.489 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:11:17.489 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:11:17.489 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_43 2023-10-11T11:11:17.490 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0 1 2 4 8 16 32 2023-10-11T11:11:17.490 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0 2023-10-11T11:11:17.490 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_image cluster2 mirror image_43 2023-10-11T11:11:17.491 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:11:17.491 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:11:17.491 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_43 2023-10-11T11:11:17.491 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_43 2023-10-11T11:11:17.536 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 rm mirror/image_43 2023-10-11T11:11:17.654 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi111.stderr:2023-10-11T11:11:17.653+0000 7f1e68fcb700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55698468a4e0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-10-11T11:11:17.654 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi111.stderr:2023-10-11T11:11:17.653+0000 7f1e6f7d8700 -1 rbd::mirror::ImageReplayer: 0x556983bd0f00 [3/008ad2e7-4a4a-41fc-aeec-300676cc4f95] operator(): start failed: (85) Interrupted system call should be restarted 2023-10-11T11:11:19.609 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-10-11T11:11:19.613 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:11:19.613 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T11:11:19.613 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_44 2023-10-11T11:11:19.613 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_snapshot cluster2 mirror image_44 snap 2023-10-11T11:11:19.614 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:11:19.614 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:11:19.614 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_44 2023-10-11T11:11:19.614 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap=snap 2023-10-11T11:11:19.614 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 snap rm mirror/image_44@snap 2023-10-11T11:11:20.456 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Removing snap: 100% complete...done. 2023-10-11T11:11:20.480 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_image_retry cluster2 mirror image_44 2023-10-11T11:11:20.481 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:11:20.481 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:11:20.481 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_44 2023-10-11T11:11:20.481 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0 1 2 4 8 16 32 2023-10-11T11:11:20.481 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0 2023-10-11T11:11:20.482 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_image cluster2 mirror image_44 2023-10-11T11:11:20.482 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:11:20.483 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:11:20.483 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_44 2023-10-11T11:11:20.483 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_44 2023-10-11T11:11:20.525 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 rm mirror/image_44 2023-10-11T11:11:20.652 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi111.stderr:2023-10-11T11:11:20.651+0000 7f1e687ca700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x556983bc5a00 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-10-11T11:11:20.652 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi111.stderr:2023-10-11T11:11:20.651+0000 7f1e6efd7700 -1 rbd::mirror::ImageReplayer: 0x5569846da780 [3/749fd0b3-e9a4-4bac-bd26-1b56437c5b8f] operator(): start failed: (85) Interrupted system call should be restarted 2023-10-11T11:11:22.610 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-10-11T11:11:22.614 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:11:22.614 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T11:11:22.614 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_45 2023-10-11T11:11:22.614 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_snapshot cluster2 mirror image_45 snap 2023-10-11T11:11:22.615 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:11:22.615 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:11:22.615 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_45 2023-10-11T11:11:22.615 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap=snap 2023-10-11T11:11:22.615 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 snap rm mirror/image_45@snap 2023-10-11T11:11:23.498 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Removing snap: 100% complete...done. 2023-10-11T11:11:23.519 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_image_retry cluster2 mirror image_45 2023-10-11T11:11:23.519 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:11:23.520 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:11:23.520 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_45 2023-10-11T11:11:23.520 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0 1 2 4 8 16 32 2023-10-11T11:11:23.520 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0 2023-10-11T11:11:23.521 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_image cluster2 mirror image_45 2023-10-11T11:11:23.521 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:11:23.521 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:11:23.521 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_45 2023-10-11T11:11:23.521 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_45 2023-10-11T11:11:23.567 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 rm mirror/image_45 2023-10-11T11:11:23.685 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi111.stderr:2023-10-11T11:11:23.684+0000 7f1e687ca700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x556983bc56c0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-10-11T11:11:23.686 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi111.stderr:2023-10-11T11:11:23.684+0000 7f1e6efd7700 -1 rbd::mirror::ImageReplayer: 0x556984a6d400 [3/fb355421-cd73-48b6-8cc6-59fefdadb677] operator(): start failed: (85) Interrupted system call should be restarted 2023-10-11T11:11:25.598 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-10-11T11:11:25.602 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:11:25.602 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T11:11:25.602 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_46 2023-10-11T11:11:25.602 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_snapshot cluster2 mirror image_46 snap 2023-10-11T11:11:25.603 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:11:25.603 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:11:25.603 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_46 2023-10-11T11:11:25.603 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap=snap 2023-10-11T11:11:25.603 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 snap rm mirror/image_46@snap 2023-10-11T11:11:26.504 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Removing snap: 100% complete...done. 2023-10-11T11:11:26.526 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_image_retry cluster2 mirror image_46 2023-10-11T11:11:26.526 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:11:26.526 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:11:26.527 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_46 2023-10-11T11:11:26.527 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0 1 2 4 8 16 32 2023-10-11T11:11:26.527 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0 2023-10-11T11:11:26.528 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_image cluster2 mirror image_46 2023-10-11T11:11:26.528 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:11:26.528 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:11:26.528 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_46 2023-10-11T11:11:26.528 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_46 2023-10-11T11:11:26.575 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 rm mirror/image_46 2023-10-11T11:11:26.705 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi111.stderr:2023-10-11T11:11:26.704+0000 7f1e687ca700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55698468ba00 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-10-11T11:11:26.705 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi111.stderr:2023-10-11T11:11:26.704+0000 7f1e6f7d8700 -1 rbd::mirror::ImageReplayer: 0x556984a06500 [3/96075109-54b3-44cf-8c55-4ffc63857fb0] operator(): start failed: (85) Interrupted system call should be restarted 2023-10-11T11:11:28.653 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-10-11T11:11:28.657 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:11:28.657 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T11:11:28.657 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_47 2023-10-11T11:11:28.657 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_snapshot cluster2 mirror image_47 snap 2023-10-11T11:11:28.658 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:11:28.658 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:11:28.658 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_47 2023-10-11T11:11:28.658 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap=snap 2023-10-11T11:11:28.659 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 snap rm mirror/image_47@snap 2023-10-11T11:11:29.510 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Removing snap: 100% complete...done. 2023-10-11T11:11:29.535 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_image_retry cluster2 mirror image_47 2023-10-11T11:11:29.535 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:11:29.535 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:11:29.536 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_47 2023-10-11T11:11:29.536 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0 1 2 4 8 16 32 2023-10-11T11:11:29.536 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0 2023-10-11T11:11:29.537 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_image cluster2 mirror image_47 2023-10-11T11:11:29.537 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:11:29.537 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:11:29.537 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_47 2023-10-11T11:11:29.538 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_47 2023-10-11T11:11:29.581 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 rm mirror/image_47 2023-10-11T11:11:29.700 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi111.stderr:2023-10-11T11:11:29.699+0000 7f1e68fcb700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x556984c7d860 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-10-11T11:11:29.711 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi111.stderr:2023-10-11T11:11:29.710+0000 7f1e68fcb700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x556984c864e0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-10-11T11:11:29.711 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi111.stderr:2023-10-11T11:11:29.710+0000 7f1e6efd7700 -1 rbd::mirror::ImageReplayer: 0x5569846e4f00 [3/76f1a297-bf03-44bd-a55f-4ceb8d92fdbd] operator(): start failed: (85) Interrupted system call should be restarted 2023-10-11T11:11:31.647 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-10-11T11:11:31.650 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:11:31.650 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T11:11:31.651 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_48 2023-10-11T11:11:31.651 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_snapshot cluster2 mirror image_48 snap 2023-10-11T11:11:31.651 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:11:31.651 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:11:31.651 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_48 2023-10-11T11:11:31.651 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap=snap 2023-10-11T11:11:31.652 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 snap rm mirror/image_48@snap 2023-10-11T11:11:32.521 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Removing snap: 100% complete...done. 2023-10-11T11:11:32.548 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_image_retry cluster2 mirror image_48 2023-10-11T11:11:32.548 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:11:32.548 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:11:32.548 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_48 2023-10-11T11:11:32.549 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0 1 2 4 8 16 32 2023-10-11T11:11:32.549 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0 2023-10-11T11:11:32.549 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_image cluster2 mirror image_48 2023-10-11T11:11:32.550 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:11:32.550 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:11:32.550 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_48 2023-10-11T11:11:32.550 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_48 2023-10-11T11:11:32.596 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 rm mirror/image_48 2023-10-11T11:11:32.724 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi111.stderr:2023-10-11T11:11:32.723+0000 7f1e687ca700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x556984c87380 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-10-11T11:11:32.724 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi111.stderr:2023-10-11T11:11:32.723+0000 7f1e6f7d8700 -1 rbd::mirror::ImageReplayer: 0x556984a07180 [3/a8b63bdf-0dc4-4ae7-bada-d821d5883be6] operator(): start failed: (85) Interrupted system call should be restarted 2023-10-11T11:11:32.752 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.5158496734532892) > weight (0.25) 2023-10-11T11:11:32.752 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.6950353904532421) > weight (0.25) 2023-10-11T11:11:32.752 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.2 2023-10-11T11:11:32.752 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Sent signal 15 2023-10-11T11:11:32.753 INFO:tasks.rbd_mirror_thrash:waiting for 1 secs before reviving daemons 2023-10-11T11:11:32.753 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi111.stderr:2023-10-11T11:11:32.753+0000 7f8895bd0700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 (PID: 124293) UID: 1000 2023-10-11T11:11:33.754 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.2 2023-10-11T11:11:33.754 INFO:teuthology.orchestra.run:waiting for 600 2023-10-11T11:11:33.755 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.2 2023-10-11T11:11:33.755 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Restarting daemon 2023-10-11T11:11:33.755 DEBUG:teuthology.orchestra.run.smithi111:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 2023-10-11T11:11:33.758 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Started 2023-10-11T11:11:33.758 INFO:tasks.rbd_mirror_thrash:waiting for 82 secs before thrashing 2023-10-11T11:11:34.657 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-10-11T11:11:34.661 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:11:34.661 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T11:11:34.661 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_49 2023-10-11T11:11:34.661 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_snapshot cluster2 mirror image_49 snap 2023-10-11T11:11:34.662 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:11:34.662 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:11:34.662 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_49 2023-10-11T11:11:34.662 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap=snap 2023-10-11T11:11:34.662 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 snap rm mirror/image_49@snap 2023-10-11T11:11:35.466 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Removing snap: 100% complete...done. 2023-10-11T11:11:35.482 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_image_retry cluster2 mirror image_49 2023-10-11T11:11:35.482 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:11:35.482 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:11:35.482 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_49 2023-10-11T11:11:35.483 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0 1 2 4 8 16 32 2023-10-11T11:11:35.483 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0 2023-10-11T11:11:35.484 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_image cluster2 mirror image_49 2023-10-11T11:11:35.484 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:11:35.484 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:11:35.484 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_49 2023-10-11T11:11:35.484 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_49 2023-10-11T11:11:35.527 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 rm mirror/image_49 2023-10-11T11:11:35.653 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi111.stderr:2023-10-11T11:11:35.652+0000 7f1e687ca700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x556984b79040 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-10-11T11:11:35.653 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi111.stderr:2023-10-11T11:11:35.652+0000 7f1e6f7d8700 -1 rbd::mirror::ImageReplayer: 0x5569846e5900 [3/80e3527d-c1f3-41a7-917b-e93369287a62] operator(): start failed: (85) Interrupted system call should be restarted 2023-10-11T11:11:37.668 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-10-11T11:11:37.671 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:11:37.672 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T11:11:37.672 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_50 2023-10-11T11:11:37.672 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_snapshot cluster2 mirror image_50 snap 2023-10-11T11:11:37.672 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:11:37.672 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:11:37.673 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_50 2023-10-11T11:11:37.673 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local snap=snap 2023-10-11T11:11:37.673 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 snap rm mirror/image_50@snap 2023-10-11T11:11:38.540 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Removing snap: 100% complete...done. 2023-10-11T11:11:38.567 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_image_retry cluster2 mirror image_50 2023-10-11T11:11:38.567 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:11:38.567 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:11:38.568 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_50 2023-10-11T11:11:38.568 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0 1 2 4 8 16 32 2023-10-11T11:11:38.568 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0 2023-10-11T11:11:38.569 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_image cluster2 mirror image_50 2023-10-11T11:11:38.569 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:11:38.569 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:11:38.569 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_50 2023-10-11T11:11:38.569 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_50 2023-10-11T11:11:38.614 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 rm mirror/image_50 2023-10-11T11:11:38.733 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi111.stderr:2023-10-11T11:11:38.732+0000 7f1e68fcb700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x556983bc4820 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-10-11T11:11:38.734 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi111.stderr:2023-10-11T11:11:38.732+0000 7f1e6efd7700 -1 rbd::mirror::ImageReplayer: 0x556984a6c280 [3/cd60b48a-16a6-4a4e-8bc0-70e977b76bac] operator(): start failed: (85) Interrupted system call should be restarted 2023-10-11T11:11:40.597 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-10-11T11:11:40.600 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:11:40.600 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ testlog 'TEST: image deletions should propagate' 2023-10-11T11:11:40.601 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ tee -a /tmp/tmp.nGgfBEp0YO/rbd-mirror.test.log 2023-10-11T11:11:40.601 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ date '+%F %T' 2023-10-11T11:11:40.602 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ echo 2023-10-11 11:11:40 TEST: image deletions should propagate 2023-10-11T11:11:40.602 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:2023-10-11 11:11:40 TEST: image deletions should propagate 2023-10-11T11:11:40.603 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_pool_images cluster1 mirror 0 2023-10-11T11:11:40.603 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:11:40.603 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:11:40.603 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image_count=0 2023-10-11T11:11:40.603 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T11:11:40.604 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local count 2023-10-11T11:11:40.604 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local last_count=0 2023-10-11T11:11:40.604 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ true 2023-10-11T11:11:40.604 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ seq 1 40 2023-10-11T11:11:40.604 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in `seq 1 40` 2023-10-11T11:11:40.604 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 1 -ne 1 2023-10-11T11:11:40.605 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ rbd --cluster cluster1 -p mirror mirror pool status 2023-10-11T11:11:40.605 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ grep 'images: ' 2023-10-11T11:11:40.606 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ cut '-d ' -f 2 2023-10-11T11:11:40.636 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd: failed to query services: (13) Permission denied 2023-10-11T11:11:40.639 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ count=0 2023-10-11T11:11:40.640 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 0 = 0 2023-10-11T11:11:40.640 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:11:40.640 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_pool_healthy cluster1 mirror 0 2023-10-11T11:11:40.640 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:11:40.640 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:11:40.641 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T11:11:40.641 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local state 2023-10-11T11:11:40.641 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ seq 1 40 2023-10-11T11:11:40.642 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in `seq 1 40` 2023-10-11T11:11:40.642 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 1 -ne 1 2023-10-11T11:11:40.642 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ rbd --cluster cluster1 -p mirror mirror pool status 2023-10-11T11:11:40.642 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ grep 'image health:' 2023-10-11T11:11:40.643 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ cut '-d ' -f 3 2023-10-11T11:11:40.674 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd: failed to query services: (13) Permission denied 2023-10-11T11:11:40.677 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ state=OK 2023-10-11T11:11:40.677 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test OK = ERROR 2023-10-11T11:11:40.677 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test OK = OK 2023-10-11T11:11:40.677 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:11:40.677 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ seq 1 50 2023-10-11T11:11:40.679 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T11:11:40.679 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_1 2023-10-11T11:11:40.679 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_present cluster1 mirror image_1 deleted 2023-10-11T11:11:40.680 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:11:40.680 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:11:40.680 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_1 2023-10-11T11:11:40.680 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local state=deleted 2023-10-11T11:11:40.680 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image_id= 2023-10-11T11:11:40.681 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T11:11:40.681 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test -n '' 2023-10-11T11:11:40.681 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ get_image_id cluster1 mirror image_1 2023-10-11T11:11:40.681 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:11:40.681 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local pool=mirror 2023-10-11T11:11:40.681 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local image=image_1 2023-10-11T11:11:40.682 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2023-10-11T11:11:40.682 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-10-11T11:11:40.911 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd: error opening image image_1: (2) No such file or directory 2023-10-11T11:11:40.914 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image_id= 2023-10-11T11:11:40.915 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-10-11T11:11:40.915 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.1 2023-10-11T11:11:41.016 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test_image_present cluster1 mirror image_1 deleted '' 2023-10-11T11:11:41.017 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:11:41.017 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:11:41.017 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_1 2023-10-11T11:11:41.017 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local test_state=deleted 2023-10-11T11:11:41.017 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image_id= 2023-10-11T11:11:41.018 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_state=deleted 2023-10-11T11:11:41.018 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_image_id 2023-10-11T11:11:41.018 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ get_image_id cluster1 mirror image_1 2023-10-11T11:11:41.018 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:11:41.018 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local pool=mirror 2023-10-11T11:11:41.019 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local image=image_1 2023-10-11T11:11:41.019 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2023-10-11T11:11:41.019 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-10-11T11:11:41.047 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd: error opening image image_1: (2) No such file or directory 2023-10-11T11:11:41.052 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ current_image_id= 2023-10-11T11:11:41.052 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test -n '' 2023-10-11T11:11:41.052 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test deleted = deleted 2023-10-11T11:11:41.052 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:11:41.052 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T11:11:41.053 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_2 2023-10-11T11:11:41.053 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_present cluster1 mirror image_2 deleted 2023-10-11T11:11:41.053 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:11:41.053 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:11:41.053 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_2 2023-10-11T11:11:41.053 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local state=deleted 2023-10-11T11:11:41.054 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image_id= 2023-10-11T11:11:41.054 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T11:11:41.054 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test -n '' 2023-10-11T11:11:41.054 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ get_image_id cluster1 mirror image_2 2023-10-11T11:11:41.054 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:11:41.055 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local pool=mirror 2023-10-11T11:11:41.055 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local image=image_2 2023-10-11T11:11:41.055 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2023-10-11T11:11:41.055 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-10-11T11:11:41.083 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd: error opening image image_2: (2) No such file or directory 2023-10-11T11:11:41.086 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image_id= 2023-10-11T11:11:41.086 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-10-11T11:11:41.086 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.1 2023-10-11T11:11:41.187 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test_image_present cluster1 mirror image_2 deleted '' 2023-10-11T11:11:41.188 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:11:41.188 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:11:41.188 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_2 2023-10-11T11:11:41.188 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local test_state=deleted 2023-10-11T11:11:41.188 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image_id= 2023-10-11T11:11:41.189 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_state=deleted 2023-10-11T11:11:41.189 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_image_id 2023-10-11T11:11:41.189 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ get_image_id cluster1 mirror image_2 2023-10-11T11:11:41.189 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:11:41.189 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local pool=mirror 2023-10-11T11:11:41.190 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local image=image_2 2023-10-11T11:11:41.190 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2023-10-11T11:11:41.190 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-10-11T11:11:41.220 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd: error opening image image_2: (2) No such file or directory 2023-10-11T11:11:41.224 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ current_image_id= 2023-10-11T11:11:41.224 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test -n '' 2023-10-11T11:11:41.224 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test deleted = deleted 2023-10-11T11:11:41.224 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:11:41.224 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T11:11:41.225 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_3 2023-10-11T11:11:41.225 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_present cluster1 mirror image_3 deleted 2023-10-11T11:11:41.225 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:11:41.225 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:11:41.225 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_3 2023-10-11T11:11:41.225 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local state=deleted 2023-10-11T11:11:41.226 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image_id= 2023-10-11T11:11:41.226 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T11:11:41.226 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test -n '' 2023-10-11T11:11:41.226 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ get_image_id cluster1 mirror image_3 2023-10-11T11:11:41.226 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:11:41.227 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local pool=mirror 2023-10-11T11:11:41.227 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local image=image_3 2023-10-11T11:11:41.227 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2023-10-11T11:11:41.227 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-10-11T11:11:41.255 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd: error opening image image_3: (2) No such file or directory 2023-10-11T11:11:41.258 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image_id= 2023-10-11T11:11:41.258 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-10-11T11:11:41.258 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.1 2023-10-11T11:11:41.360 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test_image_present cluster1 mirror image_3 deleted '' 2023-10-11T11:11:41.360 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:11:41.360 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:11:41.360 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_3 2023-10-11T11:11:41.361 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local test_state=deleted 2023-10-11T11:11:41.361 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image_id= 2023-10-11T11:11:41.361 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_state=deleted 2023-10-11T11:11:41.361 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_image_id 2023-10-11T11:11:41.361 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ get_image_id cluster1 mirror image_3 2023-10-11T11:11:41.361 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:11:41.362 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local pool=mirror 2023-10-11T11:11:41.362 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local image=image_3 2023-10-11T11:11:41.362 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2023-10-11T11:11:41.362 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-10-11T11:11:41.393 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd: error opening image image_3: (2) No such file or directory 2023-10-11T11:11:41.397 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ current_image_id= 2023-10-11T11:11:41.397 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test -n '' 2023-10-11T11:11:41.397 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test deleted = deleted 2023-10-11T11:11:41.397 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:11:41.398 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T11:11:41.398 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_4 2023-10-11T11:11:41.398 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_present cluster1 mirror image_4 deleted 2023-10-11T11:11:41.398 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:11:41.398 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:11:41.399 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_4 2023-10-11T11:11:41.399 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local state=deleted 2023-10-11T11:11:41.399 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image_id= 2023-10-11T11:11:41.399 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T11:11:41.399 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test -n '' 2023-10-11T11:11:41.401 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ get_image_id cluster1 mirror image_4 2023-10-11T11:11:41.401 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:11:41.401 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local pool=mirror 2023-10-11T11:11:41.402 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local image=image_4 2023-10-11T11:11:41.402 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2023-10-11T11:11:41.402 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-10-11T11:11:41.429 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd: error opening image image_4: (2) No such file or directory 2023-10-11T11:11:41.431 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image_id= 2023-10-11T11:11:41.432 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-10-11T11:11:41.432 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.1 2023-10-11T11:11:41.533 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test_image_present cluster1 mirror image_4 deleted '' 2023-10-11T11:11:41.533 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:11:41.533 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:11:41.534 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_4 2023-10-11T11:11:41.534 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local test_state=deleted 2023-10-11T11:11:41.534 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image_id= 2023-10-11T11:11:41.534 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_state=deleted 2023-10-11T11:11:41.534 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_image_id 2023-10-11T11:11:41.534 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ get_image_id cluster1 mirror image_4 2023-10-11T11:11:41.535 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:11:41.535 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local pool=mirror 2023-10-11T11:11:41.535 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local image=image_4 2023-10-11T11:11:41.535 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2023-10-11T11:11:41.535 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-10-11T11:11:41.565 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd: error opening image image_4: (2) No such file or directory 2023-10-11T11:11:41.569 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ current_image_id= 2023-10-11T11:11:41.569 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test -n '' 2023-10-11T11:11:41.569 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test deleted = deleted 2023-10-11T11:11:41.569 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:11:41.570 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T11:11:41.570 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_5 2023-10-11T11:11:41.570 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_present cluster1 mirror image_5 deleted 2023-10-11T11:11:41.570 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:11:41.570 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:11:41.571 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_5 2023-10-11T11:11:41.571 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local state=deleted 2023-10-11T11:11:41.571 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image_id= 2023-10-11T11:11:41.571 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T11:11:41.571 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test -n '' 2023-10-11T11:11:41.571 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ get_image_id cluster1 mirror image_5 2023-10-11T11:11:41.572 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:11:41.572 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local pool=mirror 2023-10-11T11:11:41.572 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local image=image_5 2023-10-11T11:11:41.572 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2023-10-11T11:11:41.572 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-10-11T11:11:41.600 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd: error opening image image_5: (2) No such file or directory 2023-10-11T11:11:41.603 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image_id= 2023-10-11T11:11:41.603 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-10-11T11:11:41.604 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.1 2023-10-11T11:11:41.705 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test_image_present cluster1 mirror image_5 deleted '' 2023-10-11T11:11:41.705 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:11:41.705 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:11:41.706 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_5 2023-10-11T11:11:41.706 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local test_state=deleted 2023-10-11T11:11:41.706 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image_id= 2023-10-11T11:11:41.706 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_state=deleted 2023-10-11T11:11:41.706 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_image_id 2023-10-11T11:11:41.706 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ get_image_id cluster1 mirror image_5 2023-10-11T11:11:41.707 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:11:41.707 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local pool=mirror 2023-10-11T11:11:41.707 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local image=image_5 2023-10-11T11:11:41.707 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2023-10-11T11:11:41.707 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-10-11T11:11:41.747 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd: error opening image image_5: (2) No such file or directory 2023-10-11T11:11:41.751 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ current_image_id= 2023-10-11T11:11:41.751 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test -n '' 2023-10-11T11:11:41.751 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test deleted = deleted 2023-10-11T11:11:41.751 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:11:41.752 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T11:11:41.752 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_6 2023-10-11T11:11:41.752 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_present cluster1 mirror image_6 deleted 2023-10-11T11:11:41.752 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:11:41.752 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:11:41.753 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_6 2023-10-11T11:11:41.753 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local state=deleted 2023-10-11T11:11:41.753 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image_id= 2023-10-11T11:11:41.753 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T11:11:41.753 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test -n '' 2023-10-11T11:11:41.753 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ get_image_id cluster1 mirror image_6 2023-10-11T11:11:41.754 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:11:41.754 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local pool=mirror 2023-10-11T11:11:41.754 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local image=image_6 2023-10-11T11:11:41.754 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2023-10-11T11:11:41.754 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-10-11T11:11:41.789 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd: error opening image image_6: (2) No such file or directory 2023-10-11T11:11:41.792 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image_id= 2023-10-11T11:11:41.792 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-10-11T11:11:41.792 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.1 2023-10-11T11:11:41.893 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test_image_present cluster1 mirror image_6 deleted '' 2023-10-11T11:11:41.893 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:11:41.894 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:11:41.894 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_6 2023-10-11T11:11:41.894 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local test_state=deleted 2023-10-11T11:11:41.894 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image_id= 2023-10-11T11:11:41.894 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_state=deleted 2023-10-11T11:11:41.895 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_image_id 2023-10-11T11:11:41.895 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ get_image_id cluster1 mirror image_6 2023-10-11T11:11:41.895 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:11:41.895 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local pool=mirror 2023-10-11T11:11:41.895 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local image=image_6 2023-10-11T11:11:41.895 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2023-10-11T11:11:41.896 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-10-11T11:11:41.925 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd: error opening image image_6: (2) No such file or directory 2023-10-11T11:11:41.929 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ current_image_id= 2023-10-11T11:11:41.929 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test -n '' 2023-10-11T11:11:41.930 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test deleted = deleted 2023-10-11T11:11:41.930 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:11:41.930 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T11:11:41.930 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_7 2023-10-11T11:11:41.930 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_present cluster1 mirror image_7 deleted 2023-10-11T11:11:41.931 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:11:41.931 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:11:41.931 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_7 2023-10-11T11:11:41.931 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local state=deleted 2023-10-11T11:11:41.931 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image_id= 2023-10-11T11:11:41.931 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T11:11:41.932 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test -n '' 2023-10-11T11:11:41.932 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ get_image_id cluster1 mirror image_7 2023-10-11T11:11:41.932 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:11:41.932 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local pool=mirror 2023-10-11T11:11:41.932 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local image=image_7 2023-10-11T11:11:41.932 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2023-10-11T11:11:41.933 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-10-11T11:11:41.960 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd: error opening image image_7: (2) No such file or directory 2023-10-11T11:11:41.963 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image_id= 2023-10-11T11:11:41.963 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-10-11T11:11:41.964 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.1 2023-10-11T11:11:42.065 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test_image_present cluster1 mirror image_7 deleted '' 2023-10-11T11:11:42.065 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:11:42.065 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:11:42.066 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_7 2023-10-11T11:11:42.066 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local test_state=deleted 2023-10-11T11:11:42.066 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image_id= 2023-10-11T11:11:42.066 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_state=deleted 2023-10-11T11:11:42.066 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_image_id 2023-10-11T11:11:42.067 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ get_image_id cluster1 mirror image_7 2023-10-11T11:11:42.067 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:11:42.067 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local pool=mirror 2023-10-11T11:11:42.067 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local image=image_7 2023-10-11T11:11:42.067 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2023-10-11T11:11:42.067 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-10-11T11:11:42.095 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd: error opening image image_7: (2) No such file or directory 2023-10-11T11:11:42.100 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ current_image_id= 2023-10-11T11:11:42.100 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test -n '' 2023-10-11T11:11:42.100 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test deleted = deleted 2023-10-11T11:11:42.100 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:11:42.100 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T11:11:42.101 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_8 2023-10-11T11:11:42.101 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_present cluster1 mirror image_8 deleted 2023-10-11T11:11:42.101 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:11:42.101 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:11:42.101 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_8 2023-10-11T11:11:42.101 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local state=deleted 2023-10-11T11:11:42.102 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image_id= 2023-10-11T11:11:42.102 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T11:11:42.102 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test -n '' 2023-10-11T11:11:42.102 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ get_image_id cluster1 mirror image_8 2023-10-11T11:11:42.102 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:11:42.103 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local pool=mirror 2023-10-11T11:11:42.103 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local image=image_8 2023-10-11T11:11:42.103 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2023-10-11T11:11:42.103 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-10-11T11:11:42.132 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd: error opening image image_8: (2) No such file or directory 2023-10-11T11:11:42.135 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image_id= 2023-10-11T11:11:42.135 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-10-11T11:11:42.135 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.1 2023-10-11T11:11:42.236 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test_image_present cluster1 mirror image_8 deleted '' 2023-10-11T11:11:42.236 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:11:42.237 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:11:42.237 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_8 2023-10-11T11:11:42.237 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local test_state=deleted 2023-10-11T11:11:42.237 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image_id= 2023-10-11T11:11:42.237 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_state=deleted 2023-10-11T11:11:42.238 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_image_id 2023-10-11T11:11:42.238 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ get_image_id cluster1 mirror image_8 2023-10-11T11:11:42.238 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:11:42.238 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local pool=mirror 2023-10-11T11:11:42.238 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local image=image_8 2023-10-11T11:11:42.239 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2023-10-11T11:11:42.239 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-10-11T11:11:42.270 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd: error opening image image_8: (2) No such file or directory 2023-10-11T11:11:42.274 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ current_image_id= 2023-10-11T11:11:42.274 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test -n '' 2023-10-11T11:11:42.274 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test deleted = deleted 2023-10-11T11:11:42.274 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:11:42.275 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T11:11:42.275 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_9 2023-10-11T11:11:42.275 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_present cluster1 mirror image_9 deleted 2023-10-11T11:11:42.275 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:11:42.275 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:11:42.275 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_9 2023-10-11T11:11:42.276 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local state=deleted 2023-10-11T11:11:42.276 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image_id= 2023-10-11T11:11:42.276 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T11:11:42.276 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test -n '' 2023-10-11T11:11:42.276 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ get_image_id cluster1 mirror image_9 2023-10-11T11:11:42.277 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:11:42.277 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local pool=mirror 2023-10-11T11:11:42.277 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local image=image_9 2023-10-11T11:11:42.277 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2023-10-11T11:11:42.277 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-10-11T11:11:42.306 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd: error opening image image_9: (2) No such file or directory 2023-10-11T11:11:42.309 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image_id= 2023-10-11T11:11:42.309 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-10-11T11:11:42.309 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.1 2023-10-11T11:11:42.410 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test_image_present cluster1 mirror image_9 deleted '' 2023-10-11T11:11:42.411 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:11:42.411 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:11:42.411 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_9 2023-10-11T11:11:42.411 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local test_state=deleted 2023-10-11T11:11:42.411 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image_id= 2023-10-11T11:11:42.412 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_state=deleted 2023-10-11T11:11:42.412 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_image_id 2023-10-11T11:11:42.412 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ get_image_id cluster1 mirror image_9 2023-10-11T11:11:42.412 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:11:42.412 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local pool=mirror 2023-10-11T11:11:42.413 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local image=image_9 2023-10-11T11:11:42.413 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2023-10-11T11:11:42.413 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-10-11T11:11:42.442 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd: error opening image image_9: (2) No such file or directory 2023-10-11T11:11:42.446 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ current_image_id= 2023-10-11T11:11:42.446 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test -n '' 2023-10-11T11:11:42.446 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test deleted = deleted 2023-10-11T11:11:42.446 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:11:42.447 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T11:11:42.447 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_10 2023-10-11T11:11:42.447 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_present cluster1 mirror image_10 deleted 2023-10-11T11:11:42.447 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:11:42.447 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:11:42.448 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_10 2023-10-11T11:11:42.448 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local state=deleted 2023-10-11T11:11:42.448 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image_id= 2023-10-11T11:11:42.448 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T11:11:42.448 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test -n '' 2023-10-11T11:11:42.448 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ get_image_id cluster1 mirror image_10 2023-10-11T11:11:42.449 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:11:42.449 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local pool=mirror 2023-10-11T11:11:42.449 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local image=image_10 2023-10-11T11:11:42.449 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ rbd --cluster=cluster1 info mirror/image_10 2023-10-11T11:11:42.449 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-10-11T11:11:42.477 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd: error opening image image_10: (2) No such file or directory 2023-10-11T11:11:42.480 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image_id= 2023-10-11T11:11:42.481 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-10-11T11:11:42.481 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.1 2023-10-11T11:11:42.582 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test_image_present cluster1 mirror image_10 deleted '' 2023-10-11T11:11:42.582 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:11:42.582 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:11:42.583 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_10 2023-10-11T11:11:42.583 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local test_state=deleted 2023-10-11T11:11:42.583 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image_id= 2023-10-11T11:11:42.583 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_state=deleted 2023-10-11T11:11:42.583 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_image_id 2023-10-11T11:11:42.584 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ get_image_id cluster1 mirror image_10 2023-10-11T11:11:42.584 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:11:42.584 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local pool=mirror 2023-10-11T11:11:42.584 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local image=image_10 2023-10-11T11:11:42.584 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ rbd --cluster=cluster1 info mirror/image_10 2023-10-11T11:11:42.584 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-10-11T11:11:42.614 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd: error opening image image_10: (2) No such file or directory 2023-10-11T11:11:42.618 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ current_image_id= 2023-10-11T11:11:42.618 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test -n '' 2023-10-11T11:11:42.618 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test deleted = deleted 2023-10-11T11:11:42.618 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:11:42.619 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T11:11:42.619 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_11 2023-10-11T11:11:42.619 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_present cluster1 mirror image_11 deleted 2023-10-11T11:11:42.619 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:11:42.619 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:11:42.619 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_11 2023-10-11T11:11:42.620 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local state=deleted 2023-10-11T11:11:42.620 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image_id= 2023-10-11T11:11:42.620 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T11:11:42.620 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test -n '' 2023-10-11T11:11:42.620 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ get_image_id cluster1 mirror image_11 2023-10-11T11:11:42.621 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:11:42.621 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local pool=mirror 2023-10-11T11:11:42.621 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local image=image_11 2023-10-11T11:11:42.621 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ rbd --cluster=cluster1 info mirror/image_11 2023-10-11T11:11:42.621 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-10-11T11:11:42.649 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd: error opening image image_11: (2) No such file or directory 2023-10-11T11:11:42.652 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image_id= 2023-10-11T11:11:42.652 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-10-11T11:11:42.653 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.1 2023-10-11T11:11:42.754 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test_image_present cluster1 mirror image_11 deleted '' 2023-10-11T11:11:42.754 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:11:42.754 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:11:42.754 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_11 2023-10-11T11:11:42.755 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local test_state=deleted 2023-10-11T11:11:42.755 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image_id= 2023-10-11T11:11:42.755 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_state=deleted 2023-10-11T11:11:42.755 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_image_id 2023-10-11T11:11:42.755 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ get_image_id cluster1 mirror image_11 2023-10-11T11:11:42.756 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:11:42.756 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local pool=mirror 2023-10-11T11:11:42.756 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local image=image_11 2023-10-11T11:11:42.756 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ rbd --cluster=cluster1 info mirror/image_11 2023-10-11T11:11:42.756 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-10-11T11:11:42.787 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd: error opening image image_11: (2) No such file or directory 2023-10-11T11:11:42.791 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ current_image_id= 2023-10-11T11:11:42.791 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test -n '' 2023-10-11T11:11:42.792 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test deleted = deleted 2023-10-11T11:11:42.792 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:11:42.792 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T11:11:42.792 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_12 2023-10-11T11:11:42.792 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_present cluster1 mirror image_12 deleted 2023-10-11T11:11:42.793 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:11:42.793 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:11:42.793 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_12 2023-10-11T11:11:42.793 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local state=deleted 2023-10-11T11:11:42.793 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image_id= 2023-10-11T11:11:42.793 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T11:11:42.794 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test -n '' 2023-10-11T11:11:42.794 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ get_image_id cluster1 mirror image_12 2023-10-11T11:11:42.794 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:11:42.794 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local pool=mirror 2023-10-11T11:11:42.794 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local image=image_12 2023-10-11T11:11:42.794 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ rbd --cluster=cluster1 info mirror/image_12 2023-10-11T11:11:42.795 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-10-11T11:11:42.823 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd: error opening image image_12: (2) No such file or directory 2023-10-11T11:11:42.826 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image_id= 2023-10-11T11:11:42.826 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-10-11T11:11:42.826 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.1 2023-10-11T11:11:42.928 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test_image_present cluster1 mirror image_12 deleted '' 2023-10-11T11:11:42.928 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:11:42.928 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:11:42.928 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_12 2023-10-11T11:11:42.928 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local test_state=deleted 2023-10-11T11:11:42.929 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image_id= 2023-10-11T11:11:42.929 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_state=deleted 2023-10-11T11:11:42.929 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_image_id 2023-10-11T11:11:42.929 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ get_image_id cluster1 mirror image_12 2023-10-11T11:11:42.929 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:11:42.929 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local pool=mirror 2023-10-11T11:11:42.930 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local image=image_12 2023-10-11T11:11:42.930 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ rbd --cluster=cluster1 info mirror/image_12 2023-10-11T11:11:42.930 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-10-11T11:11:42.960 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd: error opening image image_12: (2) No such file or directory 2023-10-11T11:11:42.964 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ current_image_id= 2023-10-11T11:11:42.964 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test -n '' 2023-10-11T11:11:42.964 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test deleted = deleted 2023-10-11T11:11:42.964 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:11:42.965 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T11:11:42.965 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_13 2023-10-11T11:11:42.965 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_present cluster1 mirror image_13 deleted 2023-10-11T11:11:42.965 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:11:42.965 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:11:42.965 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_13 2023-10-11T11:11:42.966 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local state=deleted 2023-10-11T11:11:42.966 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image_id= 2023-10-11T11:11:42.966 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T11:11:42.966 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test -n '' 2023-10-11T11:11:42.966 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ get_image_id cluster1 mirror image_13 2023-10-11T11:11:42.967 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:11:42.967 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local pool=mirror 2023-10-11T11:11:42.967 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local image=image_13 2023-10-11T11:11:42.967 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ rbd --cluster=cluster1 info mirror/image_13 2023-10-11T11:11:42.967 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-10-11T11:11:42.995 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd: error opening image image_13: (2) No such file or directory 2023-10-11T11:11:42.998 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image_id= 2023-10-11T11:11:42.998 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-10-11T11:11:42.998 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.1 2023-10-11T11:11:43.099 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test_image_present cluster1 mirror image_13 deleted '' 2023-10-11T11:11:43.099 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:11:43.100 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:11:43.100 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_13 2023-10-11T11:11:43.100 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local test_state=deleted 2023-10-11T11:11:43.100 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image_id= 2023-10-11T11:11:43.101 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_state=deleted 2023-10-11T11:11:43.101 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_image_id 2023-10-11T11:11:43.101 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ get_image_id cluster1 mirror image_13 2023-10-11T11:11:43.101 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:11:43.101 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local pool=mirror 2023-10-11T11:11:43.101 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local image=image_13 2023-10-11T11:11:43.102 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ rbd --cluster=cluster1 info mirror/image_13 2023-10-11T11:11:43.102 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-10-11T11:11:43.130 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd: error opening image image_13: (2) No such file or directory 2023-10-11T11:11:43.134 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ current_image_id= 2023-10-11T11:11:43.134 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test -n '' 2023-10-11T11:11:43.135 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test deleted = deleted 2023-10-11T11:11:43.135 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:11:43.135 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T11:11:43.135 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_14 2023-10-11T11:11:43.135 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_present cluster1 mirror image_14 deleted 2023-10-11T11:11:43.136 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:11:43.136 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:11:43.136 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_14 2023-10-11T11:11:43.136 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local state=deleted 2023-10-11T11:11:43.136 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image_id= 2023-10-11T11:11:43.136 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T11:11:43.137 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test -n '' 2023-10-11T11:11:43.137 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ get_image_id cluster1 mirror image_14 2023-10-11T11:11:43.137 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:11:43.137 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local pool=mirror 2023-10-11T11:11:43.137 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local image=image_14 2023-10-11T11:11:43.138 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ rbd --cluster=cluster1 info mirror/image_14 2023-10-11T11:11:43.138 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-10-11T11:11:43.165 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd: error opening image image_14: (2) No such file or directory 2023-10-11T11:11:43.168 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image_id= 2023-10-11T11:11:43.168 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-10-11T11:11:43.168 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.1 2023-10-11T11:11:43.270 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test_image_present cluster1 mirror image_14 deleted '' 2023-10-11T11:11:43.270 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:11:43.270 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:11:43.271 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_14 2023-10-11T11:11:43.271 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local test_state=deleted 2023-10-11T11:11:43.271 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image_id= 2023-10-11T11:11:43.271 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_state=deleted 2023-10-11T11:11:43.271 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_image_id 2023-10-11T11:11:43.271 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ get_image_id cluster1 mirror image_14 2023-10-11T11:11:43.272 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:11:43.272 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local pool=mirror 2023-10-11T11:11:43.272 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local image=image_14 2023-10-11T11:11:43.272 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ rbd --cluster=cluster1 info mirror/image_14 2023-10-11T11:11:43.272 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-10-11T11:11:43.301 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd: error opening image image_14: (2) No such file or directory 2023-10-11T11:11:43.305 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ current_image_id= 2023-10-11T11:11:43.305 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test -n '' 2023-10-11T11:11:43.306 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test deleted = deleted 2023-10-11T11:11:43.306 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:11:43.306 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T11:11:43.306 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_15 2023-10-11T11:11:43.306 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_present cluster1 mirror image_15 deleted 2023-10-11T11:11:43.307 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:11:43.307 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:11:43.307 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_15 2023-10-11T11:11:43.307 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local state=deleted 2023-10-11T11:11:43.307 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image_id= 2023-10-11T11:11:43.308 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T11:11:43.308 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test -n '' 2023-10-11T11:11:43.308 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ get_image_id cluster1 mirror image_15 2023-10-11T11:11:43.308 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:11:43.308 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local pool=mirror 2023-10-11T11:11:43.308 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local image=image_15 2023-10-11T11:11:43.309 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ rbd --cluster=cluster1 info mirror/image_15 2023-10-11T11:11:43.309 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-10-11T11:11:43.336 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd: error opening image image_15: (2) No such file or directory 2023-10-11T11:11:43.339 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image_id= 2023-10-11T11:11:43.340 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-10-11T11:11:43.340 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.1 2023-10-11T11:11:43.441 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test_image_present cluster1 mirror image_15 deleted '' 2023-10-11T11:11:43.441 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:11:43.441 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:11:43.442 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_15 2023-10-11T11:11:43.442 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local test_state=deleted 2023-10-11T11:11:43.442 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image_id= 2023-10-11T11:11:43.442 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_state=deleted 2023-10-11T11:11:43.442 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_image_id 2023-10-11T11:11:43.443 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ get_image_id cluster1 mirror image_15 2023-10-11T11:11:43.443 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:11:43.445 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local pool=mirror 2023-10-11T11:11:43.445 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local image=image_15 2023-10-11T11:11:43.445 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ rbd --cluster=cluster1 info mirror/image_15 2023-10-11T11:11:43.445 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-10-11T11:11:43.474 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd: error opening image image_15: (2) No such file or directory 2023-10-11T11:11:43.478 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ current_image_id= 2023-10-11T11:11:43.478 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test -n '' 2023-10-11T11:11:43.479 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test deleted = deleted 2023-10-11T11:11:43.479 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:11:43.479 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T11:11:43.479 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_16 2023-10-11T11:11:43.479 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_present cluster1 mirror image_16 deleted 2023-10-11T11:11:43.480 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:11:43.480 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:11:43.480 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_16 2023-10-11T11:11:43.480 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local state=deleted 2023-10-11T11:11:43.480 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image_id= 2023-10-11T11:11:43.480 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T11:11:43.481 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test -n '' 2023-10-11T11:11:43.481 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ get_image_id cluster1 mirror image_16 2023-10-11T11:11:43.481 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:11:43.481 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local pool=mirror 2023-10-11T11:11:43.481 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local image=image_16 2023-10-11T11:11:43.482 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ rbd --cluster=cluster1 info mirror/image_16 2023-10-11T11:11:43.482 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-10-11T11:11:43.509 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd: error opening image image_16: (2) No such file or directory 2023-10-11T11:11:43.512 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image_id= 2023-10-11T11:11:43.512 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-10-11T11:11:43.512 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.1 2023-10-11T11:11:43.614 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test_image_present cluster1 mirror image_16 deleted '' 2023-10-11T11:11:43.614 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:11:43.614 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:11:43.614 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_16 2023-10-11T11:11:43.614 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local test_state=deleted 2023-10-11T11:11:43.615 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image_id= 2023-10-11T11:11:43.615 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_state=deleted 2023-10-11T11:11:43.615 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_image_id 2023-10-11T11:11:43.615 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ get_image_id cluster1 mirror image_16 2023-10-11T11:11:43.615 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:11:43.616 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local pool=mirror 2023-10-11T11:11:43.616 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local image=image_16 2023-10-11T11:11:43.616 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ rbd --cluster=cluster1 info mirror/image_16 2023-10-11T11:11:43.616 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-10-11T11:11:43.645 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd: error opening image image_16: (2) No such file or directory 2023-10-11T11:11:43.649 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ current_image_id= 2023-10-11T11:11:43.649 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test -n '' 2023-10-11T11:11:43.649 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test deleted = deleted 2023-10-11T11:11:43.649 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:11:43.649 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T11:11:43.650 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_17 2023-10-11T11:11:43.650 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_present cluster1 mirror image_17 deleted 2023-10-11T11:11:43.650 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:11:43.650 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:11:43.650 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_17 2023-10-11T11:11:43.651 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local state=deleted 2023-10-11T11:11:43.651 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image_id= 2023-10-11T11:11:43.651 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T11:11:43.651 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test -n '' 2023-10-11T11:11:43.651 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ get_image_id cluster1 mirror image_17 2023-10-11T11:11:43.652 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:11:43.652 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local pool=mirror 2023-10-11T11:11:43.652 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local image=image_17 2023-10-11T11:11:43.652 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ rbd --cluster=cluster1 info mirror/image_17 2023-10-11T11:11:43.652 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-10-11T11:11:43.680 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd: error opening image image_17: (2) No such file or directory 2023-10-11T11:11:43.684 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image_id= 2023-10-11T11:11:43.684 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-10-11T11:11:43.684 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.1 2023-10-11T11:11:43.785 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test_image_present cluster1 mirror image_17 deleted '' 2023-10-11T11:11:43.785 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:11:43.786 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:11:43.786 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_17 2023-10-11T11:11:43.786 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local test_state=deleted 2023-10-11T11:11:43.786 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image_id= 2023-10-11T11:11:43.786 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_state=deleted 2023-10-11T11:11:43.787 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_image_id 2023-10-11T11:11:43.787 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ get_image_id cluster1 mirror image_17 2023-10-11T11:11:43.787 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:11:43.787 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local pool=mirror 2023-10-11T11:11:43.787 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local image=image_17 2023-10-11T11:11:43.788 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ rbd --cluster=cluster1 info mirror/image_17 2023-10-11T11:11:43.788 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-10-11T11:11:43.817 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd: error opening image image_17: (2) No such file or directory 2023-10-11T11:11:43.821 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ current_image_id= 2023-10-11T11:11:43.821 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test -n '' 2023-10-11T11:11:43.821 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test deleted = deleted 2023-10-11T11:11:43.821 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:11:43.822 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T11:11:43.822 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_18 2023-10-11T11:11:43.822 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_present cluster1 mirror image_18 deleted 2023-10-11T11:11:43.822 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:11:43.822 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:11:43.822 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_18 2023-10-11T11:11:43.823 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local state=deleted 2023-10-11T11:11:43.823 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image_id= 2023-10-11T11:11:43.823 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T11:11:43.823 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test -n '' 2023-10-11T11:11:43.823 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ get_image_id cluster1 mirror image_18 2023-10-11T11:11:43.824 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:11:43.824 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local pool=mirror 2023-10-11T11:11:43.824 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local image=image_18 2023-10-11T11:11:43.824 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ rbd --cluster=cluster1 info mirror/image_18 2023-10-11T11:11:43.824 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-10-11T11:11:43.852 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd: error opening image image_18: (2) No such file or directory 2023-10-11T11:11:43.855 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image_id= 2023-10-11T11:11:43.855 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-10-11T11:11:43.856 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.1 2023-10-11T11:11:43.957 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test_image_present cluster1 mirror image_18 deleted '' 2023-10-11T11:11:43.957 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:11:43.957 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:11:43.958 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_18 2023-10-11T11:11:43.958 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local test_state=deleted 2023-10-11T11:11:43.958 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image_id= 2023-10-11T11:11:43.958 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_state=deleted 2023-10-11T11:11:43.958 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_image_id 2023-10-11T11:11:43.959 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ get_image_id cluster1 mirror image_18 2023-10-11T11:11:43.959 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:11:43.959 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local pool=mirror 2023-10-11T11:11:43.959 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local image=image_18 2023-10-11T11:11:43.959 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ rbd --cluster=cluster1 info mirror/image_18 2023-10-11T11:11:43.960 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-10-11T11:11:43.989 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd: error opening image image_18: (2) No such file or directory 2023-10-11T11:11:43.993 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ current_image_id= 2023-10-11T11:11:43.993 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test -n '' 2023-10-11T11:11:43.994 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test deleted = deleted 2023-10-11T11:11:43.994 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:11:43.994 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T11:11:43.994 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_19 2023-10-11T11:11:43.994 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_present cluster1 mirror image_19 deleted 2023-10-11T11:11:43.994 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:11:43.995 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:11:43.995 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_19 2023-10-11T11:11:43.995 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local state=deleted 2023-10-11T11:11:43.995 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image_id= 2023-10-11T11:11:43.995 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T11:11:43.995 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test -n '' 2023-10-11T11:11:43.996 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ get_image_id cluster1 mirror image_19 2023-10-11T11:11:43.996 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:11:43.996 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local pool=mirror 2023-10-11T11:11:43.996 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local image=image_19 2023-10-11T11:11:43.996 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ rbd --cluster=cluster1 info mirror/image_19 2023-10-11T11:11:43.997 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-10-11T11:11:44.025 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd: error opening image image_19: (2) No such file or directory 2023-10-11T11:11:44.028 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image_id= 2023-10-11T11:11:44.029 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-10-11T11:11:44.029 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.1 2023-10-11T11:11:44.130 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test_image_present cluster1 mirror image_19 deleted '' 2023-10-11T11:11:44.130 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:11:44.130 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:11:44.130 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_19 2023-10-11T11:11:44.131 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local test_state=deleted 2023-10-11T11:11:44.131 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image_id= 2023-10-11T11:11:44.131 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_state=deleted 2023-10-11T11:11:44.131 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_image_id 2023-10-11T11:11:44.131 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ get_image_id cluster1 mirror image_19 2023-10-11T11:11:44.132 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:11:44.132 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local pool=mirror 2023-10-11T11:11:44.132 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local image=image_19 2023-10-11T11:11:44.132 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ rbd --cluster=cluster1 info mirror/image_19 2023-10-11T11:11:44.132 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-10-11T11:11:44.161 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd: error opening image image_19: (2) No such file or directory 2023-10-11T11:11:44.165 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ current_image_id= 2023-10-11T11:11:44.166 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test -n '' 2023-10-11T11:11:44.166 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test deleted = deleted 2023-10-11T11:11:44.166 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:11:44.166 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T11:11:44.167 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_20 2023-10-11T11:11:44.167 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_present cluster1 mirror image_20 deleted 2023-10-11T11:11:44.167 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:11:44.167 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:11:44.167 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_20 2023-10-11T11:11:44.167 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local state=deleted 2023-10-11T11:11:44.168 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image_id= 2023-10-11T11:11:44.168 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T11:11:44.168 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test -n '' 2023-10-11T11:11:44.168 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ get_image_id cluster1 mirror image_20 2023-10-11T11:11:44.168 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:11:44.168 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local pool=mirror 2023-10-11T11:11:44.169 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local image=image_20 2023-10-11T11:11:44.169 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ rbd --cluster=cluster1 info mirror/image_20 2023-10-11T11:11:44.169 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-10-11T11:11:44.197 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd: error opening image image_20: (2) No such file or directory 2023-10-11T11:11:44.199 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image_id= 2023-10-11T11:11:44.200 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-10-11T11:11:44.200 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.1 2023-10-11T11:11:44.301 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test_image_present cluster1 mirror image_20 deleted '' 2023-10-11T11:11:44.301 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:11:44.301 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:11:44.302 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_20 2023-10-11T11:11:44.302 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local test_state=deleted 2023-10-11T11:11:44.302 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image_id= 2023-10-11T11:11:44.302 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_state=deleted 2023-10-11T11:11:44.302 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_image_id 2023-10-11T11:11:44.303 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ get_image_id cluster1 mirror image_20 2023-10-11T11:11:44.303 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:11:44.303 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local pool=mirror 2023-10-11T11:11:44.303 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local image=image_20 2023-10-11T11:11:44.303 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ rbd --cluster=cluster1 info mirror/image_20 2023-10-11T11:11:44.304 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-10-11T11:11:44.534 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd: error opening image image_20: (2) No such file or directory 2023-10-11T11:11:44.538 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ current_image_id= 2023-10-11T11:11:44.539 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test -n '' 2023-10-11T11:11:44.539 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test deleted = deleted 2023-10-11T11:11:44.539 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:11:44.539 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T11:11:44.539 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_21 2023-10-11T11:11:44.540 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_present cluster1 mirror image_21 deleted 2023-10-11T11:11:44.540 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:11:44.540 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:11:44.540 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_21 2023-10-11T11:11:44.540 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local state=deleted 2023-10-11T11:11:44.541 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image_id= 2023-10-11T11:11:44.541 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T11:11:44.541 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test -n '' 2023-10-11T11:11:44.541 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ get_image_id cluster1 mirror image_21 2023-10-11T11:11:44.541 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:11:44.542 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local pool=mirror 2023-10-11T11:11:44.542 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local image=image_21 2023-10-11T11:11:44.542 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ rbd --cluster=cluster1 info mirror/image_21 2023-10-11T11:11:44.542 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-10-11T11:11:44.569 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd: error opening image image_21: (2) No such file or directory 2023-10-11T11:11:44.572 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image_id= 2023-10-11T11:11:44.572 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-10-11T11:11:44.573 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.1 2023-10-11T11:11:44.674 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test_image_present cluster1 mirror image_21 deleted '' 2023-10-11T11:11:44.674 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:11:44.675 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:11:44.675 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_21 2023-10-11T11:11:44.675 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local test_state=deleted 2023-10-11T11:11:44.675 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image_id= 2023-10-11T11:11:44.675 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_state=deleted 2023-10-11T11:11:44.676 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_image_id 2023-10-11T11:11:44.676 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ get_image_id cluster1 mirror image_21 2023-10-11T11:11:44.676 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:11:44.676 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local pool=mirror 2023-10-11T11:11:44.676 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local image=image_21 2023-10-11T11:11:44.676 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ rbd --cluster=cluster1 info mirror/image_21 2023-10-11T11:11:44.677 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-10-11T11:11:44.705 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd: error opening image image_21: (2) No such file or directory 2023-10-11T11:11:44.710 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ current_image_id= 2023-10-11T11:11:44.710 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test -n '' 2023-10-11T11:11:44.711 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test deleted = deleted 2023-10-11T11:11:44.711 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:11:44.711 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T11:11:44.711 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_22 2023-10-11T11:11:44.711 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_present cluster1 mirror image_22 deleted 2023-10-11T11:11:44.711 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:11:44.712 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:11:44.712 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_22 2023-10-11T11:11:44.712 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local state=deleted 2023-10-11T11:11:44.712 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image_id= 2023-10-11T11:11:44.712 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T11:11:44.713 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test -n '' 2023-10-11T11:11:44.713 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ get_image_id cluster1 mirror image_22 2023-10-11T11:11:44.713 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:11:44.713 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local pool=mirror 2023-10-11T11:11:44.713 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local image=image_22 2023-10-11T11:11:44.713 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ rbd --cluster=cluster1 info mirror/image_22 2023-10-11T11:11:44.714 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-10-11T11:11:44.741 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd: error opening image image_22: (2) No such file or directory 2023-10-11T11:11:44.744 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image_id= 2023-10-11T11:11:44.744 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-10-11T11:11:44.744 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.1 2023-10-11T11:11:44.846 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test_image_present cluster1 mirror image_22 deleted '' 2023-10-11T11:11:44.846 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:11:44.847 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:11:44.847 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_22 2023-10-11T11:11:44.847 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local test_state=deleted 2023-10-11T11:11:44.847 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image_id= 2023-10-11T11:11:44.847 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_state=deleted 2023-10-11T11:11:44.847 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_image_id 2023-10-11T11:11:44.848 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ get_image_id cluster1 mirror image_22 2023-10-11T11:11:44.848 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:11:44.848 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local pool=mirror 2023-10-11T11:11:44.848 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local image=image_22 2023-10-11T11:11:44.848 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ rbd --cluster=cluster1 info mirror/image_22 2023-10-11T11:11:44.849 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-10-11T11:11:44.877 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd: error opening image image_22: (2) No such file or directory 2023-10-11T11:11:44.881 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ current_image_id= 2023-10-11T11:11:44.882 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test -n '' 2023-10-11T11:11:44.882 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test deleted = deleted 2023-10-11T11:11:44.882 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:11:44.882 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T11:11:44.882 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_23 2023-10-11T11:11:44.883 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_present cluster1 mirror image_23 deleted 2023-10-11T11:11:44.883 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:11:44.883 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:11:44.883 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_23 2023-10-11T11:11:44.883 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local state=deleted 2023-10-11T11:11:44.884 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image_id= 2023-10-11T11:11:44.884 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T11:11:44.884 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test -n '' 2023-10-11T11:11:44.884 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ get_image_id cluster1 mirror image_23 2023-10-11T11:11:44.884 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:11:44.884 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local pool=mirror 2023-10-11T11:11:44.885 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local image=image_23 2023-10-11T11:11:44.885 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ rbd --cluster=cluster1 info mirror/image_23 2023-10-11T11:11:44.885 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-10-11T11:11:44.913 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd: error opening image image_23: (2) No such file or directory 2023-10-11T11:11:44.916 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image_id= 2023-10-11T11:11:44.917 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-10-11T11:11:44.917 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.1 2023-10-11T11:11:45.019 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test_image_present cluster1 mirror image_23 deleted '' 2023-10-11T11:11:45.019 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:11:45.019 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:11:45.019 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_23 2023-10-11T11:11:45.019 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local test_state=deleted 2023-10-11T11:11:45.020 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image_id= 2023-10-11T11:11:45.020 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_state=deleted 2023-10-11T11:11:45.020 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_image_id 2023-10-11T11:11:45.020 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ get_image_id cluster1 mirror image_23 2023-10-11T11:11:45.020 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:11:45.020 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local pool=mirror 2023-10-11T11:11:45.021 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local image=image_23 2023-10-11T11:11:45.021 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ rbd --cluster=cluster1 info mirror/image_23 2023-10-11T11:11:45.021 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-10-11T11:11:45.050 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd: error opening image image_23: (2) No such file or directory 2023-10-11T11:11:45.054 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ current_image_id= 2023-10-11T11:11:45.055 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test -n '' 2023-10-11T11:11:45.055 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test deleted = deleted 2023-10-11T11:11:45.055 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:11:45.055 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T11:11:45.055 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_24 2023-10-11T11:11:45.056 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_present cluster1 mirror image_24 deleted 2023-10-11T11:11:45.056 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:11:45.056 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:11:45.056 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_24 2023-10-11T11:11:45.056 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local state=deleted 2023-10-11T11:11:45.056 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image_id= 2023-10-11T11:11:45.057 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T11:11:45.057 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test -n '' 2023-10-11T11:11:45.057 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ get_image_id cluster1 mirror image_24 2023-10-11T11:11:45.057 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:11:45.057 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local pool=mirror 2023-10-11T11:11:45.057 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local image=image_24 2023-10-11T11:11:45.058 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ rbd --cluster=cluster1 info mirror/image_24 2023-10-11T11:11:45.058 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-10-11T11:11:45.086 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd: error opening image image_24: (2) No such file or directory 2023-10-11T11:11:45.089 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image_id= 2023-10-11T11:11:45.089 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-10-11T11:11:45.089 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.1 2023-10-11T11:11:45.191 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test_image_present cluster1 mirror image_24 deleted '' 2023-10-11T11:11:45.191 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:11:45.191 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:11:45.191 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_24 2023-10-11T11:11:45.192 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local test_state=deleted 2023-10-11T11:11:45.192 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image_id= 2023-10-11T11:11:45.192 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_state=deleted 2023-10-11T11:11:45.192 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_image_id 2023-10-11T11:11:45.192 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ get_image_id cluster1 mirror image_24 2023-10-11T11:11:45.192 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:11:45.193 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local pool=mirror 2023-10-11T11:11:45.193 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local image=image_24 2023-10-11T11:11:45.193 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ rbd --cluster=cluster1 info mirror/image_24 2023-10-11T11:11:45.193 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-10-11T11:11:45.222 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd: error opening image image_24: (2) No such file or directory 2023-10-11T11:11:45.226 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ current_image_id= 2023-10-11T11:11:45.226 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test -n '' 2023-10-11T11:11:45.227 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test deleted = deleted 2023-10-11T11:11:45.227 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:11:45.227 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T11:11:45.227 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_25 2023-10-11T11:11:45.227 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_present cluster1 mirror image_25 deleted 2023-10-11T11:11:45.227 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:11:45.228 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:11:45.228 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_25 2023-10-11T11:11:45.228 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local state=deleted 2023-10-11T11:11:45.228 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image_id= 2023-10-11T11:11:45.228 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T11:11:45.229 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test -n '' 2023-10-11T11:11:45.229 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ get_image_id cluster1 mirror image_25 2023-10-11T11:11:45.229 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:11:45.229 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local pool=mirror 2023-10-11T11:11:45.229 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local image=image_25 2023-10-11T11:11:45.229 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ rbd --cluster=cluster1 info mirror/image_25 2023-10-11T11:11:45.230 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-10-11T11:11:45.258 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd: error opening image image_25: (2) No such file or directory 2023-10-11T11:11:45.261 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image_id= 2023-10-11T11:11:45.261 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-10-11T11:11:45.261 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.1 2023-10-11T11:11:45.362 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test_image_present cluster1 mirror image_25 deleted '' 2023-10-11T11:11:45.363 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:11:45.363 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:11:45.363 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_25 2023-10-11T11:11:45.363 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local test_state=deleted 2023-10-11T11:11:45.363 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image_id= 2023-10-11T11:11:45.363 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_state=deleted 2023-10-11T11:11:45.364 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_image_id 2023-10-11T11:11:45.364 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ get_image_id cluster1 mirror image_25 2023-10-11T11:11:45.364 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:11:45.364 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local pool=mirror 2023-10-11T11:11:45.364 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local image=image_25 2023-10-11T11:11:45.364 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ rbd --cluster=cluster1 info mirror/image_25 2023-10-11T11:11:45.365 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-10-11T11:11:45.394 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd: error opening image image_25: (2) No such file or directory 2023-10-11T11:11:45.398 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ current_image_id= 2023-10-11T11:11:45.398 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test -n '' 2023-10-11T11:11:45.398 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test deleted = deleted 2023-10-11T11:11:45.398 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:11:45.399 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T11:11:45.399 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_26 2023-10-11T11:11:45.399 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_present cluster1 mirror image_26 deleted 2023-10-11T11:11:45.399 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:11:45.399 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:11:45.399 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_26 2023-10-11T11:11:45.400 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local state=deleted 2023-10-11T11:11:45.400 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image_id= 2023-10-11T11:11:45.400 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T11:11:45.400 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test -n '' 2023-10-11T11:11:45.400 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ get_image_id cluster1 mirror image_26 2023-10-11T11:11:45.400 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:11:45.401 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local pool=mirror 2023-10-11T11:11:45.401 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local image=image_26 2023-10-11T11:11:45.401 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ rbd --cluster=cluster1 info mirror/image_26 2023-10-11T11:11:45.401 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-10-11T11:11:45.428 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd: error opening image image_26: (2) No such file or directory 2023-10-11T11:11:45.432 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image_id= 2023-10-11T11:11:45.432 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-10-11T11:11:45.432 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.1 2023-10-11T11:11:45.533 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test_image_present cluster1 mirror image_26 deleted '' 2023-10-11T11:11:45.533 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:11:45.534 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:11:45.534 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_26 2023-10-11T11:11:45.534 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local test_state=deleted 2023-10-11T11:11:45.534 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image_id= 2023-10-11T11:11:45.534 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_state=deleted 2023-10-11T11:11:45.534 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_image_id 2023-10-11T11:11:45.535 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ get_image_id cluster1 mirror image_26 2023-10-11T11:11:45.535 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:11:45.535 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local pool=mirror 2023-10-11T11:11:45.535 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local image=image_26 2023-10-11T11:11:45.535 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ rbd --cluster=cluster1 info mirror/image_26 2023-10-11T11:11:45.535 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-10-11T11:11:45.565 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd: error opening image image_26: (2) No such file or directory 2023-10-11T11:11:45.569 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ current_image_id= 2023-10-11T11:11:45.569 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test -n '' 2023-10-11T11:11:45.569 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test deleted = deleted 2023-10-11T11:11:45.570 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:11:45.570 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T11:11:45.570 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_27 2023-10-11T11:11:45.570 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_present cluster1 mirror image_27 deleted 2023-10-11T11:11:45.570 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:11:45.571 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:11:45.571 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_27 2023-10-11T11:11:45.571 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local state=deleted 2023-10-11T11:11:45.571 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image_id= 2023-10-11T11:11:45.571 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T11:11:45.571 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test -n '' 2023-10-11T11:11:45.572 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ get_image_id cluster1 mirror image_27 2023-10-11T11:11:45.572 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:11:45.572 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local pool=mirror 2023-10-11T11:11:45.572 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local image=image_27 2023-10-11T11:11:45.572 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ rbd --cluster=cluster1 info mirror/image_27 2023-10-11T11:11:45.572 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-10-11T11:11:45.600 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd: error opening image image_27: (2) No such file or directory 2023-10-11T11:11:45.603 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image_id= 2023-10-11T11:11:45.604 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-10-11T11:11:45.604 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.1 2023-10-11T11:11:45.705 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test_image_present cluster1 mirror image_27 deleted '' 2023-10-11T11:11:45.705 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:11:45.705 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:11:45.706 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_27 2023-10-11T11:11:45.706 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local test_state=deleted 2023-10-11T11:11:45.706 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image_id= 2023-10-11T11:11:45.706 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_state=deleted 2023-10-11T11:11:45.706 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_image_id 2023-10-11T11:11:45.707 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ get_image_id cluster1 mirror image_27 2023-10-11T11:11:45.707 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:11:45.707 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local pool=mirror 2023-10-11T11:11:45.707 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local image=image_27 2023-10-11T11:11:45.707 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ rbd --cluster=cluster1 info mirror/image_27 2023-10-11T11:11:45.707 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-10-11T11:11:45.737 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd: error opening image image_27: (2) No such file or directory 2023-10-11T11:11:45.741 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ current_image_id= 2023-10-11T11:11:45.741 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test -n '' 2023-10-11T11:11:45.742 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test deleted = deleted 2023-10-11T11:11:45.742 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:11:45.742 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T11:11:45.742 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_28 2023-10-11T11:11:45.742 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_present cluster1 mirror image_28 deleted 2023-10-11T11:11:45.742 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:11:45.743 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:11:45.743 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_28 2023-10-11T11:11:45.743 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local state=deleted 2023-10-11T11:11:45.743 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image_id= 2023-10-11T11:11:45.743 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T11:11:45.743 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test -n '' 2023-10-11T11:11:45.744 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ get_image_id cluster1 mirror image_28 2023-10-11T11:11:45.744 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:11:45.744 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local pool=mirror 2023-10-11T11:11:45.744 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local image=image_28 2023-10-11T11:11:45.744 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ rbd --cluster=cluster1 info mirror/image_28 2023-10-11T11:11:45.744 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-10-11T11:11:45.974 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd: error opening image image_28: (2) No such file or directory 2023-10-11T11:11:45.977 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image_id= 2023-10-11T11:11:45.977 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-10-11T11:11:45.977 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.1 2023-10-11T11:11:46.078 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test_image_present cluster1 mirror image_28 deleted '' 2023-10-11T11:11:46.079 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:11:46.079 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:11:46.079 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_28 2023-10-11T11:11:46.079 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local test_state=deleted 2023-10-11T11:11:46.079 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image_id= 2023-10-11T11:11:46.079 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_state=deleted 2023-10-11T11:11:46.080 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_image_id 2023-10-11T11:11:46.080 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ get_image_id cluster1 mirror image_28 2023-10-11T11:11:46.080 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:11:46.080 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local pool=mirror 2023-10-11T11:11:46.080 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local image=image_28 2023-10-11T11:11:46.080 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ rbd --cluster=cluster1 info mirror/image_28 2023-10-11T11:11:46.081 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-10-11T11:11:46.110 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd: error opening image image_28: (2) No such file or directory 2023-10-11T11:11:46.114 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ current_image_id= 2023-10-11T11:11:46.115 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test -n '' 2023-10-11T11:11:46.115 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test deleted = deleted 2023-10-11T11:11:46.115 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:11:46.115 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T11:11:46.115 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_29 2023-10-11T11:11:46.115 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_present cluster1 mirror image_29 deleted 2023-10-11T11:11:46.116 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:11:46.116 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:11:46.116 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_29 2023-10-11T11:11:46.116 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local state=deleted 2023-10-11T11:11:46.116 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image_id= 2023-10-11T11:11:46.116 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T11:11:46.117 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test -n '' 2023-10-11T11:11:46.117 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ get_image_id cluster1 mirror image_29 2023-10-11T11:11:46.117 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:11:46.117 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local pool=mirror 2023-10-11T11:11:46.117 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local image=image_29 2023-10-11T11:11:46.117 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ rbd --cluster=cluster1 info mirror/image_29 2023-10-11T11:11:46.118 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-10-11T11:11:46.146 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd: error opening image image_29: (2) No such file or directory 2023-10-11T11:11:46.149 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image_id= 2023-10-11T11:11:46.149 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-10-11T11:11:46.149 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.1 2023-10-11T11:11:46.251 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test_image_present cluster1 mirror image_29 deleted '' 2023-10-11T11:11:46.251 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:11:46.251 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:11:46.251 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_29 2023-10-11T11:11:46.251 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local test_state=deleted 2023-10-11T11:11:46.252 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image_id= 2023-10-11T11:11:46.252 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_state=deleted 2023-10-11T11:11:46.252 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_image_id 2023-10-11T11:11:46.252 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ get_image_id cluster1 mirror image_29 2023-10-11T11:11:46.252 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:11:46.252 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local pool=mirror 2023-10-11T11:11:46.253 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local image=image_29 2023-10-11T11:11:46.253 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ rbd --cluster=cluster1 info mirror/image_29 2023-10-11T11:11:46.253 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-10-11T11:11:46.281 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd: error opening image image_29: (2) No such file or directory 2023-10-11T11:11:46.285 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ current_image_id= 2023-10-11T11:11:46.286 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test -n '' 2023-10-11T11:11:46.286 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test deleted = deleted 2023-10-11T11:11:46.286 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:11:46.286 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T11:11:46.286 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_30 2023-10-11T11:11:46.287 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_present cluster1 mirror image_30 deleted 2023-10-11T11:11:46.287 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:11:46.287 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:11:46.287 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_30 2023-10-11T11:11:46.287 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local state=deleted 2023-10-11T11:11:46.287 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image_id= 2023-10-11T11:11:46.288 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T11:11:46.288 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test -n '' 2023-10-11T11:11:46.288 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ get_image_id cluster1 mirror image_30 2023-10-11T11:11:46.288 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:11:46.288 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local pool=mirror 2023-10-11T11:11:46.288 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local image=image_30 2023-10-11T11:11:46.289 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ rbd --cluster=cluster1 info mirror/image_30 2023-10-11T11:11:46.289 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-10-11T11:11:46.317 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd: error opening image image_30: (2) No such file or directory 2023-10-11T11:11:46.320 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image_id= 2023-10-11T11:11:46.320 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-10-11T11:11:46.320 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.1 2023-10-11T11:11:46.421 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test_image_present cluster1 mirror image_30 deleted '' 2023-10-11T11:11:46.422 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:11:46.422 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:11:46.422 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_30 2023-10-11T11:11:46.422 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local test_state=deleted 2023-10-11T11:11:46.423 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image_id= 2023-10-11T11:11:46.423 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_state=deleted 2023-10-11T11:11:46.423 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_image_id 2023-10-11T11:11:46.423 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ get_image_id cluster1 mirror image_30 2023-10-11T11:11:46.423 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:11:46.423 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local pool=mirror 2023-10-11T11:11:46.424 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local image=image_30 2023-10-11T11:11:46.424 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ rbd --cluster=cluster1 info mirror/image_30 2023-10-11T11:11:46.424 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-10-11T11:11:46.453 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd: error opening image image_30: (2) No such file or directory 2023-10-11T11:11:46.457 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ current_image_id= 2023-10-11T11:11:46.457 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test -n '' 2023-10-11T11:11:46.457 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test deleted = deleted 2023-10-11T11:11:46.458 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:11:46.458 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T11:11:46.458 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_31 2023-10-11T11:11:46.458 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_present cluster1 mirror image_31 deleted 2023-10-11T11:11:46.458 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:11:46.458 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:11:46.459 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_31 2023-10-11T11:11:46.459 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local state=deleted 2023-10-11T11:11:46.459 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image_id= 2023-10-11T11:11:46.459 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T11:11:46.459 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test -n '' 2023-10-11T11:11:46.459 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ get_image_id cluster1 mirror image_31 2023-10-11T11:11:46.460 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:11:46.460 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local pool=mirror 2023-10-11T11:11:46.460 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local image=image_31 2023-10-11T11:11:46.460 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ rbd --cluster=cluster1 info mirror/image_31 2023-10-11T11:11:46.460 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-10-11T11:11:46.489 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd: error opening image image_31: (2) No such file or directory 2023-10-11T11:11:46.491 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image_id= 2023-10-11T11:11:46.492 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-10-11T11:11:46.492 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.1 2023-10-11T11:11:46.593 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test_image_present cluster1 mirror image_31 deleted '' 2023-10-11T11:11:46.593 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:11:46.594 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:11:46.594 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_31 2023-10-11T11:11:46.594 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local test_state=deleted 2023-10-11T11:11:46.594 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image_id= 2023-10-11T11:11:46.594 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_state=deleted 2023-10-11T11:11:46.595 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_image_id 2023-10-11T11:11:46.595 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ get_image_id cluster1 mirror image_31 2023-10-11T11:11:46.595 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:11:46.595 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local pool=mirror 2023-10-11T11:11:46.595 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local image=image_31 2023-10-11T11:11:46.595 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ rbd --cluster=cluster1 info mirror/image_31 2023-10-11T11:11:46.595 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-10-11T11:11:46.625 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd: error opening image image_31: (2) No such file or directory 2023-10-11T11:11:46.629 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ current_image_id= 2023-10-11T11:11:46.630 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test -n '' 2023-10-11T11:11:46.630 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test deleted = deleted 2023-10-11T11:11:46.630 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:11:46.630 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T11:11:46.630 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_32 2023-10-11T11:11:46.631 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_present cluster1 mirror image_32 deleted 2023-10-11T11:11:46.631 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:11:46.631 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:11:46.631 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_32 2023-10-11T11:11:46.631 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local state=deleted 2023-10-11T11:11:46.631 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image_id= 2023-10-11T11:11:46.632 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T11:11:46.632 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test -n '' 2023-10-11T11:11:46.632 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ get_image_id cluster1 mirror image_32 2023-10-11T11:11:46.632 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:11:46.632 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local pool=mirror 2023-10-11T11:11:46.632 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local image=image_32 2023-10-11T11:11:46.633 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ rbd --cluster=cluster1 info mirror/image_32 2023-10-11T11:11:46.633 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-10-11T11:11:46.661 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd: error opening image image_32: (2) No such file or directory 2023-10-11T11:11:46.664 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image_id= 2023-10-11T11:11:46.664 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-10-11T11:11:46.664 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.1 2023-10-11T11:11:46.766 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test_image_present cluster1 mirror image_32 deleted '' 2023-10-11T11:11:46.766 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:11:46.766 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:11:46.766 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_32 2023-10-11T11:11:46.766 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local test_state=deleted 2023-10-11T11:11:46.767 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image_id= 2023-10-11T11:11:46.767 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_state=deleted 2023-10-11T11:11:46.767 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_image_id 2023-10-11T11:11:46.767 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ get_image_id cluster1 mirror image_32 2023-10-11T11:11:46.767 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:11:46.767 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local pool=mirror 2023-10-11T11:11:46.768 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local image=image_32 2023-10-11T11:11:46.768 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ rbd --cluster=cluster1 info mirror/image_32 2023-10-11T11:11:46.768 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-10-11T11:11:46.798 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd: error opening image image_32: (2) No such file or directory 2023-10-11T11:11:46.802 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ current_image_id= 2023-10-11T11:11:46.802 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test -n '' 2023-10-11T11:11:46.802 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test deleted = deleted 2023-10-11T11:11:46.803 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:11:46.803 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T11:11:46.803 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_33 2023-10-11T11:11:46.803 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_present cluster1 mirror image_33 deleted 2023-10-11T11:11:46.803 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:11:46.803 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:11:46.804 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_33 2023-10-11T11:11:46.804 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local state=deleted 2023-10-11T11:11:46.804 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image_id= 2023-10-11T11:11:46.804 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T11:11:46.804 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test -n '' 2023-10-11T11:11:46.804 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ get_image_id cluster1 mirror image_33 2023-10-11T11:11:46.805 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:11:46.805 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local pool=mirror 2023-10-11T11:11:46.805 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local image=image_33 2023-10-11T11:11:46.805 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ rbd --cluster=cluster1 info mirror/image_33 2023-10-11T11:11:46.805 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-10-11T11:11:46.833 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd: error opening image image_33: (2) No such file or directory 2023-10-11T11:11:46.836 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image_id= 2023-10-11T11:11:46.836 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-10-11T11:11:46.836 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.1 2023-10-11T11:11:46.938 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test_image_present cluster1 mirror image_33 deleted '' 2023-10-11T11:11:46.938 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:11:46.938 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:11:46.939 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_33 2023-10-11T11:11:46.939 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local test_state=deleted 2023-10-11T11:11:46.939 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image_id= 2023-10-11T11:11:46.939 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_state=deleted 2023-10-11T11:11:46.939 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_image_id 2023-10-11T11:11:46.939 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ get_image_id cluster1 mirror image_33 2023-10-11T11:11:46.940 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:11:46.940 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local pool=mirror 2023-10-11T11:11:46.940 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local image=image_33 2023-10-11T11:11:46.940 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ rbd --cluster=cluster1 info mirror/image_33 2023-10-11T11:11:46.940 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-10-11T11:11:46.969 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd: error opening image image_33: (2) No such file or directory 2023-10-11T11:11:46.973 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ current_image_id= 2023-10-11T11:11:46.973 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test -n '' 2023-10-11T11:11:46.973 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test deleted = deleted 2023-10-11T11:11:46.973 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:11:46.974 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T11:11:46.974 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_34 2023-10-11T11:11:46.974 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_present cluster1 mirror image_34 deleted 2023-10-11T11:11:46.974 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:11:46.974 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:11:46.975 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_34 2023-10-11T11:11:46.975 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local state=deleted 2023-10-11T11:11:46.975 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image_id= 2023-10-11T11:11:46.975 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T11:11:46.975 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test -n '' 2023-10-11T11:11:46.975 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ get_image_id cluster1 mirror image_34 2023-10-11T11:11:46.976 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:11:46.976 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local pool=mirror 2023-10-11T11:11:46.976 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local image=image_34 2023-10-11T11:11:46.976 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ rbd --cluster=cluster1 info mirror/image_34 2023-10-11T11:11:46.976 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-10-11T11:11:47.005 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd: error opening image image_34: (2) No such file or directory 2023-10-11T11:11:47.008 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image_id= 2023-10-11T11:11:47.008 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-10-11T11:11:47.008 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.1 2023-10-11T11:11:47.110 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test_image_present cluster1 mirror image_34 deleted '' 2023-10-11T11:11:47.110 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:11:47.110 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:11:47.110 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_34 2023-10-11T11:11:47.110 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local test_state=deleted 2023-10-11T11:11:47.111 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image_id= 2023-10-11T11:11:47.111 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_state=deleted 2023-10-11T11:11:47.111 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_image_id 2023-10-11T11:11:47.111 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ get_image_id cluster1 mirror image_34 2023-10-11T11:11:47.111 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:11:47.111 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local pool=mirror 2023-10-11T11:11:47.112 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local image=image_34 2023-10-11T11:11:47.112 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ rbd --cluster=cluster1 info mirror/image_34 2023-10-11T11:11:47.112 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-10-11T11:11:47.141 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd: error opening image image_34: (2) No such file or directory 2023-10-11T11:11:47.145 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ current_image_id= 2023-10-11T11:11:47.146 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test -n '' 2023-10-11T11:11:47.146 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test deleted = deleted 2023-10-11T11:11:47.146 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:11:47.146 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T11:11:47.146 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_35 2023-10-11T11:11:47.147 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_present cluster1 mirror image_35 deleted 2023-10-11T11:11:47.147 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:11:47.147 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:11:47.147 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_35 2023-10-11T11:11:47.147 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local state=deleted 2023-10-11T11:11:47.148 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image_id= 2023-10-11T11:11:47.148 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T11:11:47.148 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test -n '' 2023-10-11T11:11:47.148 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ get_image_id cluster1 mirror image_35 2023-10-11T11:11:47.148 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:11:47.148 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local pool=mirror 2023-10-11T11:11:47.149 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local image=image_35 2023-10-11T11:11:47.149 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ rbd --cluster=cluster1 info mirror/image_35 2023-10-11T11:11:47.149 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-10-11T11:11:47.177 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd: error opening image image_35: (2) No such file or directory 2023-10-11T11:11:47.180 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image_id= 2023-10-11T11:11:47.180 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-10-11T11:11:47.180 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.1 2023-10-11T11:11:47.282 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test_image_present cluster1 mirror image_35 deleted '' 2023-10-11T11:11:47.282 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:11:47.282 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:11:47.282 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_35 2023-10-11T11:11:47.282 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local test_state=deleted 2023-10-11T11:11:47.283 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image_id= 2023-10-11T11:11:47.283 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_state=deleted 2023-10-11T11:11:47.283 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_image_id 2023-10-11T11:11:47.283 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ get_image_id cluster1 mirror image_35 2023-10-11T11:11:47.283 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:11:47.283 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local pool=mirror 2023-10-11T11:11:47.284 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local image=image_35 2023-10-11T11:11:47.284 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ rbd --cluster=cluster1 info mirror/image_35 2023-10-11T11:11:47.284 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-10-11T11:11:47.314 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd: error opening image image_35: (2) No such file or directory 2023-10-11T11:11:47.318 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ current_image_id= 2023-10-11T11:11:47.318 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test -n '' 2023-10-11T11:11:47.318 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test deleted = deleted 2023-10-11T11:11:47.318 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:11:47.319 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T11:11:47.319 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_36 2023-10-11T11:11:47.319 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_present cluster1 mirror image_36 deleted 2023-10-11T11:11:47.319 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:11:47.319 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:11:47.319 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_36 2023-10-11T11:11:47.320 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local state=deleted 2023-10-11T11:11:47.320 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image_id= 2023-10-11T11:11:47.320 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T11:11:47.320 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test -n '' 2023-10-11T11:11:47.320 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ get_image_id cluster1 mirror image_36 2023-10-11T11:11:47.320 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:11:47.321 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local pool=mirror 2023-10-11T11:11:47.321 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local image=image_36 2023-10-11T11:11:47.321 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ rbd --cluster=cluster1 info mirror/image_36 2023-10-11T11:11:47.321 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-10-11T11:11:47.350 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd: error opening image image_36: (2) No such file or directory 2023-10-11T11:11:47.353 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image_id= 2023-10-11T11:11:47.353 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-10-11T11:11:47.353 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.1 2023-10-11T11:11:47.455 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test_image_present cluster1 mirror image_36 deleted '' 2023-10-11T11:11:47.455 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:11:47.455 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:11:47.455 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_36 2023-10-11T11:11:47.456 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local test_state=deleted 2023-10-11T11:11:47.456 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image_id= 2023-10-11T11:11:47.456 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_state=deleted 2023-10-11T11:11:47.456 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_image_id 2023-10-11T11:11:47.456 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ get_image_id cluster1 mirror image_36 2023-10-11T11:11:47.457 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:11:47.457 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local pool=mirror 2023-10-11T11:11:47.457 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local image=image_36 2023-10-11T11:11:47.457 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ rbd --cluster=cluster1 info mirror/image_36 2023-10-11T11:11:47.457 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-10-11T11:11:47.486 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd: error opening image image_36: (2) No such file or directory 2023-10-11T11:11:47.491 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ current_image_id= 2023-10-11T11:11:47.491 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test -n '' 2023-10-11T11:11:47.492 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test deleted = deleted 2023-10-11T11:11:47.492 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:11:47.492 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T11:11:47.492 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_37 2023-10-11T11:11:47.492 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_present cluster1 mirror image_37 deleted 2023-10-11T11:11:47.493 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:11:47.493 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:11:47.493 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_37 2023-10-11T11:11:47.493 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local state=deleted 2023-10-11T11:11:47.493 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image_id= 2023-10-11T11:11:47.494 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T11:11:47.494 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test -n '' 2023-10-11T11:11:47.494 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ get_image_id cluster1 mirror image_37 2023-10-11T11:11:47.494 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:11:47.494 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local pool=mirror 2023-10-11T11:11:47.495 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local image=image_37 2023-10-11T11:11:47.495 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ rbd --cluster=cluster1 info mirror/image_37 2023-10-11T11:11:47.495 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-10-11T11:11:47.522 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd: error opening image image_37: (2) No such file or directory 2023-10-11T11:11:47.525 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image_id= 2023-10-11T11:11:47.525 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-10-11T11:11:47.526 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.1 2023-10-11T11:11:47.627 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test_image_present cluster1 mirror image_37 deleted '' 2023-10-11T11:11:47.627 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:11:47.627 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:11:47.628 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_37 2023-10-11T11:11:47.628 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local test_state=deleted 2023-10-11T11:11:47.628 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image_id= 2023-10-11T11:11:47.628 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_state=deleted 2023-10-11T11:11:47.628 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_image_id 2023-10-11T11:11:47.629 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ get_image_id cluster1 mirror image_37 2023-10-11T11:11:47.629 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:11:47.629 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local pool=mirror 2023-10-11T11:11:47.629 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local image=image_37 2023-10-11T11:11:47.629 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ rbd --cluster=cluster1 info mirror/image_37 2023-10-11T11:11:47.630 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-10-11T11:11:47.659 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd: error opening image image_37: (2) No such file or directory 2023-10-11T11:11:47.663 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ current_image_id= 2023-10-11T11:11:47.664 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test -n '' 2023-10-11T11:11:47.664 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test deleted = deleted 2023-10-11T11:11:47.664 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:11:47.664 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T11:11:47.664 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_38 2023-10-11T11:11:47.665 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_present cluster1 mirror image_38 deleted 2023-10-11T11:11:47.665 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:11:47.665 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:11:47.665 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_38 2023-10-11T11:11:47.665 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local state=deleted 2023-10-11T11:11:47.666 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image_id= 2023-10-11T11:11:47.666 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T11:11:47.666 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test -n '' 2023-10-11T11:11:47.666 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ get_image_id cluster1 mirror image_38 2023-10-11T11:11:47.666 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:11:47.667 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local pool=mirror 2023-10-11T11:11:47.667 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local image=image_38 2023-10-11T11:11:47.667 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ rbd --cluster=cluster1 info mirror/image_38 2023-10-11T11:11:47.667 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-10-11T11:11:47.694 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd: error opening image image_38: (2) No such file or directory 2023-10-11T11:11:47.697 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image_id= 2023-10-11T11:11:47.697 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-10-11T11:11:47.697 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.1 2023-10-11T11:11:47.799 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test_image_present cluster1 mirror image_38 deleted '' 2023-10-11T11:11:47.799 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:11:47.799 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:11:47.800 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_38 2023-10-11T11:11:47.800 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local test_state=deleted 2023-10-11T11:11:47.800 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image_id= 2023-10-11T11:11:47.800 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_state=deleted 2023-10-11T11:11:47.800 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_image_id 2023-10-11T11:11:47.801 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ get_image_id cluster1 mirror image_38 2023-10-11T11:11:47.801 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:11:47.801 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local pool=mirror 2023-10-11T11:11:47.801 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local image=image_38 2023-10-11T11:11:47.801 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ rbd --cluster=cluster1 info mirror/image_38 2023-10-11T11:11:47.802 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-10-11T11:11:47.830 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd: error opening image image_38: (2) No such file or directory 2023-10-11T11:11:47.834 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ current_image_id= 2023-10-11T11:11:47.834 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test -n '' 2023-10-11T11:11:47.835 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test deleted = deleted 2023-10-11T11:11:47.835 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:11:47.835 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T11:11:47.835 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_39 2023-10-11T11:11:47.835 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_present cluster1 mirror image_39 deleted 2023-10-11T11:11:47.836 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:11:47.836 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:11:47.836 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_39 2023-10-11T11:11:47.836 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local state=deleted 2023-10-11T11:11:47.836 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image_id= 2023-10-11T11:11:47.837 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T11:11:47.837 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test -n '' 2023-10-11T11:11:47.837 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ get_image_id cluster1 mirror image_39 2023-10-11T11:11:47.837 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:11:47.837 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local pool=mirror 2023-10-11T11:11:47.838 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local image=image_39 2023-10-11T11:11:47.838 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ rbd --cluster=cluster1 info mirror/image_39 2023-10-11T11:11:47.838 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-10-11T11:11:47.865 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd: error opening image image_39: (2) No such file or directory 2023-10-11T11:11:47.868 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image_id= 2023-10-11T11:11:47.869 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-10-11T11:11:47.869 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.1 2023-10-11T11:11:47.971 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test_image_present cluster1 mirror image_39 deleted '' 2023-10-11T11:11:47.971 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:11:47.971 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:11:47.971 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_39 2023-10-11T11:11:47.972 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local test_state=deleted 2023-10-11T11:11:47.972 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image_id= 2023-10-11T11:11:47.972 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_state=deleted 2023-10-11T11:11:47.972 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_image_id 2023-10-11T11:11:47.972 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ get_image_id cluster1 mirror image_39 2023-10-11T11:11:47.972 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:11:47.973 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local pool=mirror 2023-10-11T11:11:47.973 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local image=image_39 2023-10-11T11:11:47.973 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ rbd --cluster=cluster1 info mirror/image_39 2023-10-11T11:11:47.973 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-10-11T11:11:48.003 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd: error opening image image_39: (2) No such file or directory 2023-10-11T11:11:48.008 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ current_image_id= 2023-10-11T11:11:48.008 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test -n '' 2023-10-11T11:11:48.008 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test deleted = deleted 2023-10-11T11:11:48.008 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:11:48.008 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T11:11:48.009 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_40 2023-10-11T11:11:48.009 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_present cluster1 mirror image_40 deleted 2023-10-11T11:11:48.009 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:11:48.009 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:11:48.009 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_40 2023-10-11T11:11:48.010 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local state=deleted 2023-10-11T11:11:48.010 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image_id= 2023-10-11T11:11:48.010 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T11:11:48.010 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test -n '' 2023-10-11T11:11:48.010 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ get_image_id cluster1 mirror image_40 2023-10-11T11:11:48.011 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:11:48.011 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local pool=mirror 2023-10-11T11:11:48.011 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local image=image_40 2023-10-11T11:11:48.011 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ rbd --cluster=cluster1 info mirror/image_40 2023-10-11T11:11:48.011 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-10-11T11:11:48.039 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd: error opening image image_40: (2) No such file or directory 2023-10-11T11:11:48.042 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image_id= 2023-10-11T11:11:48.042 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-10-11T11:11:48.042 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.1 2023-10-11T11:11:48.143 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test_image_present cluster1 mirror image_40 deleted '' 2023-10-11T11:11:48.144 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:11:48.144 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:11:48.144 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_40 2023-10-11T11:11:48.144 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local test_state=deleted 2023-10-11T11:11:48.144 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image_id= 2023-10-11T11:11:48.145 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_state=deleted 2023-10-11T11:11:48.145 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_image_id 2023-10-11T11:11:48.145 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ get_image_id cluster1 mirror image_40 2023-10-11T11:11:48.145 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:11:48.145 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local pool=mirror 2023-10-11T11:11:48.146 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local image=image_40 2023-10-11T11:11:48.146 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ rbd --cluster=cluster1 info mirror/image_40 2023-10-11T11:11:48.146 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-10-11T11:11:48.177 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd: error opening image image_40: (2) No such file or directory 2023-10-11T11:11:48.182 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ current_image_id= 2023-10-11T11:11:48.182 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test -n '' 2023-10-11T11:11:48.182 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test deleted = deleted 2023-10-11T11:11:48.183 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:11:48.183 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T11:11:48.183 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_41 2023-10-11T11:11:48.183 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_present cluster1 mirror image_41 deleted 2023-10-11T11:11:48.184 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:11:48.184 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:11:48.184 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_41 2023-10-11T11:11:48.184 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local state=deleted 2023-10-11T11:11:48.184 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image_id= 2023-10-11T11:11:48.185 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T11:11:48.185 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test -n '' 2023-10-11T11:11:48.185 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ get_image_id cluster1 mirror image_41 2023-10-11T11:11:48.185 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:11:48.185 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local pool=mirror 2023-10-11T11:11:48.186 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local image=image_41 2023-10-11T11:11:48.186 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ rbd --cluster=cluster1 info mirror/image_41 2023-10-11T11:11:48.186 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-10-11T11:11:48.213 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd: error opening image image_41: (2) No such file or directory 2023-10-11T11:11:48.216 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image_id= 2023-10-11T11:11:48.216 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-10-11T11:11:48.216 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.1 2023-10-11T11:11:48.318 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test_image_present cluster1 mirror image_41 deleted '' 2023-10-11T11:11:48.318 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:11:48.319 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:11:48.319 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_41 2023-10-11T11:11:48.319 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local test_state=deleted 2023-10-11T11:11:48.319 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image_id= 2023-10-11T11:11:48.319 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_state=deleted 2023-10-11T11:11:48.320 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_image_id 2023-10-11T11:11:48.320 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ get_image_id cluster1 mirror image_41 2023-10-11T11:11:48.320 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:11:48.320 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local pool=mirror 2023-10-11T11:11:48.320 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local image=image_41 2023-10-11T11:11:48.321 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ rbd --cluster=cluster1 info mirror/image_41 2023-10-11T11:11:48.321 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-10-11T11:11:48.351 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd: error opening image image_41: (2) No such file or directory 2023-10-11T11:11:48.355 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ current_image_id= 2023-10-11T11:11:48.355 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test -n '' 2023-10-11T11:11:48.356 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test deleted = deleted 2023-10-11T11:11:48.356 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:11:48.356 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T11:11:48.356 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_42 2023-10-11T11:11:48.356 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_present cluster1 mirror image_42 deleted 2023-10-11T11:11:48.357 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:11:48.357 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:11:48.357 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_42 2023-10-11T11:11:48.357 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local state=deleted 2023-10-11T11:11:48.357 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image_id= 2023-10-11T11:11:48.358 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T11:11:48.358 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test -n '' 2023-10-11T11:11:48.358 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ get_image_id cluster1 mirror image_42 2023-10-11T11:11:48.358 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:11:48.358 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local pool=mirror 2023-10-11T11:11:48.359 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local image=image_42 2023-10-11T11:11:48.359 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ rbd --cluster=cluster1 info mirror/image_42 2023-10-11T11:11:48.359 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-10-11T11:11:48.386 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd: error opening image image_42: (2) No such file or directory 2023-10-11T11:11:48.389 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image_id= 2023-10-11T11:11:48.389 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-10-11T11:11:48.389 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.1 2023-10-11T11:11:48.491 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test_image_present cluster1 mirror image_42 deleted '' 2023-10-11T11:11:48.491 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:11:48.491 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:11:48.491 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_42 2023-10-11T11:11:48.492 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local test_state=deleted 2023-10-11T11:11:48.492 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image_id= 2023-10-11T11:11:48.492 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_state=deleted 2023-10-11T11:11:48.492 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_image_id 2023-10-11T11:11:48.492 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ get_image_id cluster1 mirror image_42 2023-10-11T11:11:48.492 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:11:48.493 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local pool=mirror 2023-10-11T11:11:48.493 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local image=image_42 2023-10-11T11:11:48.493 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ rbd --cluster=cluster1 info mirror/image_42 2023-10-11T11:11:48.493 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-10-11T11:11:48.523 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd: error opening image image_42: (2) No such file or directory 2023-10-11T11:11:48.527 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ current_image_id= 2023-10-11T11:11:48.527 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test -n '' 2023-10-11T11:11:48.527 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test deleted = deleted 2023-10-11T11:11:48.527 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:11:48.528 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T11:11:48.528 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_43 2023-10-11T11:11:48.528 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_present cluster1 mirror image_43 deleted 2023-10-11T11:11:48.528 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:11:48.528 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:11:48.529 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_43 2023-10-11T11:11:48.529 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local state=deleted 2023-10-11T11:11:48.529 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image_id= 2023-10-11T11:11:48.529 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T11:11:48.529 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test -n '' 2023-10-11T11:11:48.529 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ get_image_id cluster1 mirror image_43 2023-10-11T11:11:48.530 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:11:48.530 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local pool=mirror 2023-10-11T11:11:48.530 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local image=image_43 2023-10-11T11:11:48.530 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ rbd --cluster=cluster1 info mirror/image_43 2023-10-11T11:11:48.530 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-10-11T11:11:48.558 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd: error opening image image_43: (2) No such file or directory 2023-10-11T11:11:48.561 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image_id= 2023-10-11T11:11:48.561 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-10-11T11:11:48.561 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.1 2023-10-11T11:11:48.663 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test_image_present cluster1 mirror image_43 deleted '' 2023-10-11T11:11:48.663 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:11:48.664 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:11:48.664 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_43 2023-10-11T11:11:48.664 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local test_state=deleted 2023-10-11T11:11:48.664 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image_id= 2023-10-11T11:11:48.664 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_state=deleted 2023-10-11T11:11:48.664 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_image_id 2023-10-11T11:11:48.665 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ get_image_id cluster1 mirror image_43 2023-10-11T11:11:48.665 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:11:48.665 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local pool=mirror 2023-10-11T11:11:48.665 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local image=image_43 2023-10-11T11:11:48.665 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ rbd --cluster=cluster1 info mirror/image_43 2023-10-11T11:11:48.666 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-10-11T11:11:48.695 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd: error opening image image_43: (2) No such file or directory 2023-10-11T11:11:48.699 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ current_image_id= 2023-10-11T11:11:48.699 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test -n '' 2023-10-11T11:11:48.699 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test deleted = deleted 2023-10-11T11:11:48.699 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:11:48.700 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T11:11:48.700 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_44 2023-10-11T11:11:48.700 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_present cluster1 mirror image_44 deleted 2023-10-11T11:11:48.700 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:11:48.700 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:11:48.701 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_44 2023-10-11T11:11:48.701 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local state=deleted 2023-10-11T11:11:48.701 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image_id= 2023-10-11T11:11:48.701 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T11:11:48.701 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test -n '' 2023-10-11T11:11:48.702 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ get_image_id cluster1 mirror image_44 2023-10-11T11:11:48.702 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:11:48.702 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local pool=mirror 2023-10-11T11:11:48.702 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local image=image_44 2023-10-11T11:11:48.702 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ rbd --cluster=cluster1 info mirror/image_44 2023-10-11T11:11:48.702 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-10-11T11:11:48.729 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd: error opening image image_44: (2) No such file or directory 2023-10-11T11:11:48.732 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image_id= 2023-10-11T11:11:48.733 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-10-11T11:11:48.733 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.1 2023-10-11T11:11:48.835 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test_image_present cluster1 mirror image_44 deleted '' 2023-10-11T11:11:48.835 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:11:48.835 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:11:48.835 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_44 2023-10-11T11:11:48.836 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local test_state=deleted 2023-10-11T11:11:48.836 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image_id= 2023-10-11T11:11:48.836 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_state=deleted 2023-10-11T11:11:48.836 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_image_id 2023-10-11T11:11:48.836 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ get_image_id cluster1 mirror image_44 2023-10-11T11:11:48.837 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:11:48.837 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local pool=mirror 2023-10-11T11:11:48.837 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local image=image_44 2023-10-11T11:11:48.837 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ rbd --cluster=cluster1 info mirror/image_44 2023-10-11T11:11:48.837 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-10-11T11:11:48.866 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd: error opening image image_44: (2) No such file or directory 2023-10-11T11:11:48.869 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ current_image_id= 2023-10-11T11:11:48.870 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test -n '' 2023-10-11T11:11:48.870 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test deleted = deleted 2023-10-11T11:11:48.870 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:11:48.870 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T11:11:48.871 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_45 2023-10-11T11:11:48.871 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_present cluster1 mirror image_45 deleted 2023-10-11T11:11:48.871 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:11:48.871 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:11:48.871 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_45 2023-10-11T11:11:48.872 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local state=deleted 2023-10-11T11:11:48.872 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image_id= 2023-10-11T11:11:48.872 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T11:11:48.872 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test -n '' 2023-10-11T11:11:48.872 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ get_image_id cluster1 mirror image_45 2023-10-11T11:11:48.873 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:11:48.873 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local pool=mirror 2023-10-11T11:11:48.873 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local image=image_45 2023-10-11T11:11:48.873 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ rbd --cluster=cluster1 info mirror/image_45 2023-10-11T11:11:48.873 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-10-11T11:11:48.901 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd: error opening image image_45: (2) No such file or directory 2023-10-11T11:11:48.904 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image_id= 2023-10-11T11:11:48.904 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-10-11T11:11:48.904 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.1 2023-10-11T11:11:49.006 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test_image_present cluster1 mirror image_45 deleted '' 2023-10-11T11:11:49.006 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:11:49.006 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:11:49.006 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_45 2023-10-11T11:11:49.006 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local test_state=deleted 2023-10-11T11:11:49.007 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image_id= 2023-10-11T11:11:49.007 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_state=deleted 2023-10-11T11:11:49.007 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_image_id 2023-10-11T11:11:49.007 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ get_image_id cluster1 mirror image_45 2023-10-11T11:11:49.007 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:11:49.008 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local pool=mirror 2023-10-11T11:11:49.008 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local image=image_45 2023-10-11T11:11:49.008 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ rbd --cluster=cluster1 info mirror/image_45 2023-10-11T11:11:49.008 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-10-11T11:11:49.038 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd: error opening image image_45: (2) No such file or directory 2023-10-11T11:11:49.042 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ current_image_id= 2023-10-11T11:11:49.042 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test -n '' 2023-10-11T11:11:49.042 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test deleted = deleted 2023-10-11T11:11:49.042 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:11:49.042 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T11:11:49.043 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_46 2023-10-11T11:11:49.043 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_present cluster1 mirror image_46 deleted 2023-10-11T11:11:49.043 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:11:49.043 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:11:49.043 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_46 2023-10-11T11:11:49.044 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local state=deleted 2023-10-11T11:11:49.044 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image_id= 2023-10-11T11:11:49.044 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T11:11:49.044 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test -n '' 2023-10-11T11:11:49.044 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ get_image_id cluster1 mirror image_46 2023-10-11T11:11:49.044 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:11:49.045 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local pool=mirror 2023-10-11T11:11:49.045 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local image=image_46 2023-10-11T11:11:49.045 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ rbd --cluster=cluster1 info mirror/image_46 2023-10-11T11:11:49.045 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-10-11T11:11:49.073 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd: error opening image image_46: (2) No such file or directory 2023-10-11T11:11:49.076 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image_id= 2023-10-11T11:11:49.077 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-10-11T11:11:49.077 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.1 2023-10-11T11:11:49.178 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test_image_present cluster1 mirror image_46 deleted '' 2023-10-11T11:11:49.179 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:11:49.179 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:11:49.179 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_46 2023-10-11T11:11:49.179 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local test_state=deleted 2023-10-11T11:11:49.180 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image_id= 2023-10-11T11:11:49.180 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_state=deleted 2023-10-11T11:11:49.180 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_image_id 2023-10-11T11:11:49.180 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ get_image_id cluster1 mirror image_46 2023-10-11T11:11:49.180 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:11:49.181 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local pool=mirror 2023-10-11T11:11:49.181 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local image=image_46 2023-10-11T11:11:49.181 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ rbd --cluster=cluster1 info mirror/image_46 2023-10-11T11:11:49.181 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-10-11T11:11:49.210 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd: error opening image image_46: (2) No such file or directory 2023-10-11T11:11:49.214 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ current_image_id= 2023-10-11T11:11:49.214 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test -n '' 2023-10-11T11:11:49.215 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test deleted = deleted 2023-10-11T11:11:49.215 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:11:49.215 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T11:11:49.215 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_47 2023-10-11T11:11:49.215 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_present cluster1 mirror image_47 deleted 2023-10-11T11:11:49.216 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:11:49.216 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:11:49.216 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_47 2023-10-11T11:11:49.216 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local state=deleted 2023-10-11T11:11:49.216 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image_id= 2023-10-11T11:11:49.217 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T11:11:49.217 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test -n '' 2023-10-11T11:11:49.217 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ get_image_id cluster1 mirror image_47 2023-10-11T11:11:49.217 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:11:49.217 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local pool=mirror 2023-10-11T11:11:49.218 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local image=image_47 2023-10-11T11:11:49.218 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ rbd --cluster=cluster1 info mirror/image_47 2023-10-11T11:11:49.218 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-10-11T11:11:49.245 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd: error opening image image_47: (2) No such file or directory 2023-10-11T11:11:49.248 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image_id= 2023-10-11T11:11:49.248 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-10-11T11:11:49.248 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.1 2023-10-11T11:11:49.349 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test_image_present cluster1 mirror image_47 deleted '' 2023-10-11T11:11:49.350 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:11:49.350 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:11:49.350 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_47 2023-10-11T11:11:49.350 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local test_state=deleted 2023-10-11T11:11:49.350 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image_id= 2023-10-11T11:11:49.351 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_state=deleted 2023-10-11T11:11:49.351 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_image_id 2023-10-11T11:11:49.351 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ get_image_id cluster1 mirror image_47 2023-10-11T11:11:49.351 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:11:49.351 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local pool=mirror 2023-10-11T11:11:49.352 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local image=image_47 2023-10-11T11:11:49.352 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ rbd --cluster=cluster1 info mirror/image_47 2023-10-11T11:11:49.352 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-10-11T11:11:49.381 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd: error opening image image_47: (2) No such file or directory 2023-10-11T11:11:49.385 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ current_image_id= 2023-10-11T11:11:49.385 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test -n '' 2023-10-11T11:11:49.385 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test deleted = deleted 2023-10-11T11:11:49.385 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:11:49.386 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T11:11:49.386 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_48 2023-10-11T11:11:49.386 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_present cluster1 mirror image_48 deleted 2023-10-11T11:11:49.386 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:11:49.386 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:11:49.387 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_48 2023-10-11T11:11:49.387 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local state=deleted 2023-10-11T11:11:49.387 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image_id= 2023-10-11T11:11:49.387 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T11:11:49.387 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test -n '' 2023-10-11T11:11:49.388 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ get_image_id cluster1 mirror image_48 2023-10-11T11:11:49.388 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:11:49.388 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local pool=mirror 2023-10-11T11:11:49.388 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local image=image_48 2023-10-11T11:11:49.388 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ rbd --cluster=cluster1 info mirror/image_48 2023-10-11T11:11:49.388 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-10-11T11:11:49.416 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd: error opening image image_48: (2) No such file or directory 2023-10-11T11:11:49.419 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image_id= 2023-10-11T11:11:49.419 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-10-11T11:11:49.419 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.1 2023-10-11T11:11:49.521 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test_image_present cluster1 mirror image_48 deleted '' 2023-10-11T11:11:49.521 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:11:49.521 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:11:49.521 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_48 2023-10-11T11:11:49.522 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local test_state=deleted 2023-10-11T11:11:49.522 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image_id= 2023-10-11T11:11:49.522 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_state=deleted 2023-10-11T11:11:49.522 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_image_id 2023-10-11T11:11:49.522 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ get_image_id cluster1 mirror image_48 2023-10-11T11:11:49.523 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:11:49.523 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local pool=mirror 2023-10-11T11:11:49.523 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local image=image_48 2023-10-11T11:11:49.523 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ rbd --cluster=cluster1 info mirror/image_48 2023-10-11T11:11:49.523 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-10-11T11:11:49.554 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd: error opening image image_48: (2) No such file or directory 2023-10-11T11:11:49.558 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ current_image_id= 2023-10-11T11:11:49.558 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test -n '' 2023-10-11T11:11:49.558 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test deleted = deleted 2023-10-11T11:11:49.559 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:11:49.559 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T11:11:49.559 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_49 2023-10-11T11:11:49.559 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_present cluster1 mirror image_49 deleted 2023-10-11T11:11:49.559 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:11:49.560 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:11:49.560 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_49 2023-10-11T11:11:49.560 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local state=deleted 2023-10-11T11:11:49.560 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image_id= 2023-10-11T11:11:49.560 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T11:11:49.560 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test -n '' 2023-10-11T11:11:49.561 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ get_image_id cluster1 mirror image_49 2023-10-11T11:11:49.561 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:11:49.561 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local pool=mirror 2023-10-11T11:11:49.561 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local image=image_49 2023-10-11T11:11:49.561 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ rbd --cluster=cluster1 info mirror/image_49 2023-10-11T11:11:49.562 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-10-11T11:11:49.588 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd: error opening image image_49: (2) No such file or directory 2023-10-11T11:11:49.591 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image_id= 2023-10-11T11:11:49.592 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-10-11T11:11:49.592 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.1 2023-10-11T11:11:49.694 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test_image_present cluster1 mirror image_49 deleted '' 2023-10-11T11:11:49.694 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:11:49.695 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:11:49.695 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_49 2023-10-11T11:11:49.695 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local test_state=deleted 2023-10-11T11:11:49.696 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image_id= 2023-10-11T11:11:49.696 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_state=deleted 2023-10-11T11:11:49.696 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_image_id 2023-10-11T11:11:49.697 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ get_image_id cluster1 mirror image_49 2023-10-11T11:11:49.697 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:11:49.697 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local pool=mirror 2023-10-11T11:11:49.698 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local image=image_49 2023-10-11T11:11:49.698 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ rbd --cluster=cluster1 info mirror/image_49 2023-10-11T11:11:49.699 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-10-11T11:11:49.724 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd: error opening image image_49: (2) No such file or directory 2023-10-11T11:11:49.729 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ current_image_id= 2023-10-11T11:11:49.729 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test -n '' 2023-10-11T11:11:49.730 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test deleted = deleted 2023-10-11T11:11:49.730 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:11:49.730 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-10-11T11:11:49.731 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_50 2023-10-11T11:11:49.731 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_present cluster1 mirror image_50 deleted 2023-10-11T11:11:49.731 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:11:49.732 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:11:49.732 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_50 2023-10-11T11:11:49.732 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local state=deleted 2023-10-11T11:11:49.733 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image_id= 2023-10-11T11:11:49.733 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T11:11:49.733 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test -n '' 2023-10-11T11:11:49.734 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ get_image_id cluster1 mirror image_50 2023-10-11T11:11:49.734 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:11:49.734 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local pool=mirror 2023-10-11T11:11:49.735 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local image=image_50 2023-10-11T11:11:49.735 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ rbd --cluster=cluster1 info mirror/image_50 2023-10-11T11:11:49.735 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-10-11T11:11:49.760 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd: error opening image image_50: (2) No such file or directory 2023-10-11T11:11:49.763 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image_id= 2023-10-11T11:11:49.764 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-10-11T11:11:49.764 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.1 2023-10-11T11:11:49.866 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test_image_present cluster1 mirror image_50 deleted '' 2023-10-11T11:11:49.867 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:11:49.867 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:11:49.867 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_50 2023-10-11T11:11:49.867 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local test_state=deleted 2023-10-11T11:11:49.868 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image_id= 2023-10-11T11:11:49.868 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_state=deleted 2023-10-11T11:11:49.868 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_image_id 2023-10-11T11:11:49.869 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ get_image_id cluster1 mirror image_50 2023-10-11T11:11:49.869 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:11:49.869 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local pool=mirror 2023-10-11T11:11:49.870 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local image=image_50 2023-10-11T11:11:49.870 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ rbd --cluster=cluster1 info mirror/image_50 2023-10-11T11:11:49.870 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-10-11T11:11:49.897 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd: error opening image image_50: (2) No such file or directory 2023-10-11T11:11:49.902 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ current_image_id= 2023-10-11T11:11:49.903 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test -n '' 2023-10-11T11:11:49.903 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test deleted = deleted 2023-10-11T11:11:49.903 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:11:49.904 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ testlog 'TEST: delete images during bootstrap' 2023-10-11T11:11:49.904 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ tee -a /tmp/tmp.nGgfBEp0YO/rbd-mirror.test.log 2023-10-11T11:11:49.904 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ date '+%F %T' 2023-10-11T11:11:49.905 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ echo 2023-10-11 11:11:49 TEST: delete images during bootstrap 2023-10-11T11:11:49.905 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:2023-10-11 11:11:49 TEST: delete images during bootstrap 2023-10-11T11:11:49.905 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ set_pool_mirror_mode cluster1 mirror image 2023-10-11T11:11:49.906 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:11:49.906 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:11:49.906 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local mode=image 2023-10-11T11:11:49.906 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster1 mirror pool enable mirror image 2023-10-11T11:11:49.933 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:rbd: mirroring is already configured for image mode 2023-10-11T11:11:49.936 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ set_pool_mirror_mode cluster2 mirror image 2023-10-11T11:11:49.937 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:11:49.937 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:11:49.937 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local mode=image 2023-10-11T11:11:49.938 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 mirror pool enable mirror image 2023-10-11T11:11:49.967 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:rbd: mirroring is already configured for image mode 2023-10-11T11:11:49.971 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ start_mirror cluster1 2023-10-11T11:11:49.972 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:11:49.972 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local instance 2023-10-11T11:11:49.972 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ set_cluster_instance cluster1 cluster instance 2023-10-11T11:11:49.972 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local val=cluster1 2023-10-11T11:11:49.973 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster_var_name=cluster 2023-10-11T11:11:49.973 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local instance_var_name=instance 2023-10-11T11:11:49.973 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ cluster=cluster1 2023-10-11T11:11:49.973 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ instance=cluster1 2023-10-11T11:11:49.973 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' cluster1 = cluster1 ']' 2023-10-11T11:11:49.973 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ instance=0 2023-10-11T11:11:49.974 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ eval cluster=cluster1 2023-10-11T11:11:49.974 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ cluster=cluster1 2023-10-11T11:11:49.974 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ eval instance=0 2023-10-11T11:11:49.974 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ instance=0 2023-10-11T11:11:49.975 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test -n 1 2023-10-11T11:11:49.975 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 2023-10-11T11:11:49.975 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=test 2023-10-11T11:11:49.975 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ seq 1 10 2023-10-11T11:11:49.975 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 10` 2023-10-11T11:11:49.976 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_1 2023-10-11T11:11:49.976 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ create_image cluster2 mirror image_1 512M 2023-10-11T11:11:49.976 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:11:49.976 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T11:11:49.976 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:11:49.976 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T11:11:49.977 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_1 2023-10-11T11:11:49.977 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T11:11:49.977 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local size=128 2023-10-11T11:11:49.977 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n 512M ']' 2023-10-11T11:11:49.977 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ size=512M 2023-10-11T11:11:49.978 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T11:11:49.978 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_1 2023-10-11T11:11:50.023 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ enable_mirror cluster2 mirror image_1 2023-10-11T11:11:50.024 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:11:50.024 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:11:50.024 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_1 2023-10-11T11:11:50.024 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local mode=snapshot 2023-10-11T11:11:50.024 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_1 snapshot 2023-10-11T11:11:50.489 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:Mirroring enabled 2023-10-11T11:11:50.499 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 info mirror/image_1 2023-10-11T11:11:50.534 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:rbd image 'image_1': 2023-10-11T11:11:50.534 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: size 512 MiB in 128 objects 2023-10-11T11:11:50.534 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: order 22 (4 MiB objects) 2023-10-11T11:11:50.534 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: snapshot_count: 1 2023-10-11T11:11:50.535 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: id: 18ac86f195c3 2023-10-11T11:11:50.535 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: block_name_prefix: rbd_data.18ac86f195c3 2023-10-11T11:11:50.535 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: format: 2 2023-10-11T11:11:50.535 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-10-11T11:11:50.535 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: op_features: 2023-10-11T11:11:50.536 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: flags: 2023-10-11T11:11:50.536 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: create_timestamp: Wed Oct 11 11:11:50 2023 2023-10-11T11:11:50.536 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: access_timestamp: Wed Oct 11 11:11:50 2023 2023-10-11T11:11:50.536 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: modify_timestamp: Wed Oct 11 11:11:50 2023 2023-10-11T11:11:50.536 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring state: enabled 2023-10-11T11:11:50.536 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring mode: snapshot 2023-10-11T11:11:50.537 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring global id: af4707e5-92e2-48f5-bf31-d1f0e6818e38 2023-10-11T11:11:50.537 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring primary: true 2023-10-11T11:11:50.538 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ stress_write_image cluster2 mirror image_1 2023-10-11T11:11:50.538 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:11:50.538 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:11:50.538 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_1 2023-10-11T11:11:50.538 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-10-11T11:11:50.539 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local duration=7 2023-10-11T11:11:50.539 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ set +e 2023-10-11T11:11:50.539 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_1 --debug-rbd=20 --debug-journaler=20 2023-10-11T11:11:50.788 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:100 IOs, 10296320 bytes 2023-10-11T11:11:50.944 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:200 IOs, 13158400 bytes 2023-10-11T11:11:51.194 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:300 IOs, 85339136 bytes 2023-10-11T11:11:51.368 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:400 IOs, 138205184 bytes 2023-10-11T11:11:51.584 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:500 IOs, 181494784 bytes 2023-10-11T11:11:51.724 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:600 IOs, 203948032 bytes 2023-10-11T11:11:51.946 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:700 IOs, 262677504 bytes 2023-10-11T11:11:52.085 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:800 IOs, 276356096 bytes 2023-10-11T11:11:52.188 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:900 IOs, 276968448 bytes 2023-10-11T11:11:52.471 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1000 IOs, 370627584 bytes 2023-10-11T11:11:52.707 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1100 IOs, 430351360 bytes 2023-10-11T11:11:52.867 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1200 IOs, 461765632 bytes 2023-10-11T11:11:53.144 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1300 IOs, 541879296 bytes 2023-10-11T11:11:53.308 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1400 IOs, 577455104 bytes 2023-10-11T11:11:53.451 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1500 IOs, 604350464 bytes 2023-10-11T11:11:53.637 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1600 IOs, 626892800 bytes 2023-10-11T11:11:53.899 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1700 IOs, 702559232 bytes 2023-10-11T11:11:54.122 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1800 IOs, 754314240 bytes 2023-10-11T11:11:54.344 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1900 IOs, 823172096 bytes 2023-10-11T11:11:54.475 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2000 IOs, 825603072 bytes 2023-10-11T11:11:54.756 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2100 IOs, 919305216 bytes 2023-10-11T11:11:54.962 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2200 IOs, 982702080 bytes 2023-10-11T11:11:55.145 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2300 IOs, 1002710016 bytes 2023-10-11T11:11:55.300 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2400 IOs, 1024963584 bytes 2023-10-11T11:11:55.467 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2500 IOs, 1054779392 bytes 2023-10-11T11:11:55.655 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2600 IOs, 1104310272 bytes 2023-10-11T11:11:55.937 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2700 IOs, 1196555264 bytes 2023-10-11T11:11:56.088 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2800 IOs, 1218658304 bytes 2023-10-11T11:11:56.253 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2900 IOs, 1245257728 bytes 2023-10-11T11:11:56.400 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3000 IOs, 1263211520 bytes 2023-10-11T11:11:56.504 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3100 IOs, 1263814656 bytes 2023-10-11T11:11:56.655 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3200 IOs, 1284511744 bytes 2023-10-11T11:11:56.939 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3300 IOs, 1398166528 bytes 2023-10-11T11:11:57.111 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3400 IOs, 1414834176 bytes 2023-10-11T11:11:57.299 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3500 IOs, 1449853952 bytes 2023-10-11T11:11:57.469 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3600 IOs, 1481544704 bytes 2023-10-11T11:11:57.545 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ error_code=124 2023-10-11T11:11:57.546 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ set -e 2023-10-11T11:11:57.546 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' 124 -eq 124 ']' 2023-10-11T11:11:57.546 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:11:57.546 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_present cluster1 mirror image_1 present 2023-10-11T11:11:57.546 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:11:57.547 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:11:57.547 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_1 2023-10-11T11:11:57.547 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local state=present 2023-10-11T11:11:57.547 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image_id= 2023-10-11T11:11:57.547 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T11:11:57.547 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test -n '' 2023-10-11T11:11:57.548 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ get_image_id cluster1 mirror image_1 2023-10-11T11:11:57.548 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:11:57.548 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local pool=mirror 2023-10-11T11:11:57.548 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local image=image_1 2023-10-11T11:11:57.548 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2023-10-11T11:11:57.549 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-10-11T11:11:57.588 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image_id=13168c423a4 2023-10-11T11:11:57.588 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-10-11T11:11:57.589 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.1 2023-10-11T11:11:57.690 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test_image_present cluster1 mirror image_1 present 13168c423a4 2023-10-11T11:11:57.691 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:11:57.691 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:11:57.691 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_1 2023-10-11T11:11:57.691 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local test_state=present 2023-10-11T11:11:57.691 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image_id=13168c423a4 2023-10-11T11:11:57.692 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_state=deleted 2023-10-11T11:11:57.692 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_image_id 2023-10-11T11:11:57.692 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ get_image_id cluster1 mirror image_1 2023-10-11T11:11:57.692 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:11:57.692 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local pool=mirror 2023-10-11T11:11:57.693 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local image=image_1 2023-10-11T11:11:57.693 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2023-10-11T11:11:57.693 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-10-11T11:11:57.732 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ current_image_id=13168c423a4 2023-10-11T11:11:57.732 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test -n 13168c423a4 2023-10-11T11:11:57.732 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test -z 13168c423a4 -o 13168c423a4 = 13168c423a4 2023-10-11T11:11:57.733 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ current_state=present 2023-10-11T11:11:57.733 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test present = present 2023-10-11T11:11:57.733 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:11:57.733 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ disable_mirror cluster2 mirror image_1 2023-10-11T11:11:57.733 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:11:57.734 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:11:57.734 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_1 2023-10-11T11:11:57.734 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_1 2023-10-11T11:11:59.700 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi111.stderr:2023-10-11T11:11:59.699+0000 7f1e68fcb700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x556984c7c680 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-10-11T11:11:59.701 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi111.stderr:2023-10-11T11:11:59.699+0000 7f1e6f7d8700 -1 rbd::mirror::ImageReplayer: 0x556984a6c280 [3/af4707e5-92e2-48f5-bf31-d1f0e6818e38] operator(): start failed: (85) Interrupted system call should be restarted 2023-10-11T11:12:28.600 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi111.stderr:2023-10-11T11:12:28.599+0000 7f2c296d2700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55c1e94f6400 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=368 2023-10-11T11:12:28.601 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi111.stderr:2023-10-11T11:12:28.599+0000 7f2c296d2700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55c1e94f6400 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=372, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=9a182971-2e50-451f-a839-7c08d3b6157c, primary_snap_id=170, last_copied_object_number=128, snap_seqs={368=18446744073709551614}] 2023-10-11T11:12:28.604 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:Mirroring disabled 2023-10-11T11:12:28.627 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_present cluster1 mirror image_1 deleted 2023-10-11T11:12:28.627 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:12:28.627 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:12:28.628 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_1 2023-10-11T11:12:28.628 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local state=deleted 2023-10-11T11:12:28.628 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image_id= 2023-10-11T11:12:28.628 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T11:12:28.628 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test -n '' 2023-10-11T11:12:28.629 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ get_image_id cluster1 mirror image_1 2023-10-11T11:12:28.629 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:12:28.629 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local pool=mirror 2023-10-11T11:12:28.629 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local image=image_1 2023-10-11T11:12:28.629 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2023-10-11T11:12:28.630 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-10-11T11:12:28.662 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd: error opening image image_1: (2) No such file or directory 2023-10-11T11:12:28.665 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image_id= 2023-10-11T11:12:28.666 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-10-11T11:12:28.666 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.1 2023-10-11T11:12:28.767 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test_image_present cluster1 mirror image_1 deleted '' 2023-10-11T11:12:28.768 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:12:28.768 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:12:28.768 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_1 2023-10-11T11:12:28.768 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local test_state=deleted 2023-10-11T11:12:28.769 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image_id= 2023-10-11T11:12:28.769 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_state=deleted 2023-10-11T11:12:28.769 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_image_id 2023-10-11T11:12:28.769 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ get_image_id cluster1 mirror image_1 2023-10-11T11:12:28.769 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:12:28.770 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local pool=mirror 2023-10-11T11:12:28.770 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local image=image_1 2023-10-11T11:12:28.770 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2023-10-11T11:12:28.770 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-10-11T11:12:28.813 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd: error opening image image_1: (2) No such file or directory 2023-10-11T11:12:28.816 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ current_image_id= 2023-10-11T11:12:28.816 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test -n '' 2023-10-11T11:12:28.817 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test deleted = deleted 2023-10-11T11:12:28.817 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:12:28.817 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ purge_snapshots cluster2 mirror image_1 2023-10-11T11:12:28.817 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:12:28.818 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:12:28.818 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_1 2023-10-11T11:12:28.818 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 snap purge mirror/image_1 2023-10-11T11:12:28.860 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_image_retry cluster2 mirror image_1 2023-10-11T11:12:28.861 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:12:28.861 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:12:28.861 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_1 2023-10-11T11:12:28.861 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0 1 2 4 8 16 32 2023-10-11T11:12:28.862 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0 2023-10-11T11:12:28.863 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_image cluster2 mirror image_1 2023-10-11T11:12:28.863 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:12:28.863 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:12:28.863 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_1 2023-10-11T11:12:28.863 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_1 2023-10-11T11:12:28.904 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 rm mirror/image_1 2023-10-11T11:12:29.185 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2023-10-11T11:12:29.189 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:12:29.190 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 10` 2023-10-11T11:12:29.190 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_2 2023-10-11T11:12:29.190 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ create_image cluster2 mirror image_2 512M 2023-10-11T11:12:29.190 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:12:29.190 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T11:12:29.191 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:12:29.191 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T11:12:29.191 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_2 2023-10-11T11:12:29.191 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T11:12:29.191 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local size=128 2023-10-11T11:12:29.192 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n 512M ']' 2023-10-11T11:12:29.192 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ size=512M 2023-10-11T11:12:29.192 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T11:12:29.192 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_2 2023-10-11T11:12:29.238 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ enable_mirror cluster2 mirror image_2 2023-10-11T11:12:29.239 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:12:29.239 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:12:29.239 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_2 2023-10-11T11:12:29.239 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local mode=snapshot 2023-10-11T11:12:29.240 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_2 snapshot 2023-10-11T11:12:29.610 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:Mirroring enabled 2023-10-11T11:12:29.620 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 info mirror/image_2 2023-10-11T11:12:29.656 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:rbd image 'image_2': 2023-10-11T11:12:29.657 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: size 512 MiB in 128 objects 2023-10-11T11:12:29.657 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: order 22 (4 MiB objects) 2023-10-11T11:12:29.657 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: snapshot_count: 1 2023-10-11T11:12:29.657 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: id: 18c448887886 2023-10-11T11:12:29.658 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: block_name_prefix: rbd_data.18c448887886 2023-10-11T11:12:29.658 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: format: 2 2023-10-11T11:12:29.658 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-10-11T11:12:29.658 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: op_features: 2023-10-11T11:12:29.659 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: flags: 2023-10-11T11:12:29.659 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: create_timestamp: Wed Oct 11 11:12:29 2023 2023-10-11T11:12:29.659 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: access_timestamp: Wed Oct 11 11:12:29 2023 2023-10-11T11:12:29.660 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: modify_timestamp: Wed Oct 11 11:12:29 2023 2023-10-11T11:12:29.660 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring state: enabled 2023-10-11T11:12:29.660 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring mode: snapshot 2023-10-11T11:12:29.660 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring global id: 20741d1a-dcdc-4270-a299-73cd05081476 2023-10-11T11:12:29.661 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring primary: true 2023-10-11T11:12:29.662 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ stress_write_image cluster2 mirror image_2 2023-10-11T11:12:29.662 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:12:29.663 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:12:29.663 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_2 2023-10-11T11:12:29.663 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-10-11T11:12:29.663 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local duration=13 2023-10-11T11:12:29.664 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ set +e 2023-10-11T11:12:29.664 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_2 --debug-rbd=20 --debug-journaler=20 2023-10-11T11:12:29.928 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:100 IOs, 16492544 bytes 2023-10-11T11:12:30.123 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:200 IOs, 49078272 bytes 2023-10-11T11:12:30.268 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:300 IOs, 62842880 bytes 2023-10-11T11:12:30.408 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:400 IOs, 80921600 bytes 2023-10-11T11:12:30.572 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:500 IOs, 109940736 bytes 2023-10-11T11:12:30.806 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:600 IOs, 190970880 bytes 2023-10-11T11:12:30.969 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:700 IOs, 214647808 bytes 2023-10-11T11:12:31.239 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:800 IOs, 295165952 bytes 2023-10-11T11:12:31.480 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:900 IOs, 385385472 bytes 2023-10-11T11:12:31.688 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1000 IOs, 422316032 bytes 2023-10-11T11:12:31.950 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1100 IOs, 512645120 bytes 2023-10-11T11:12:32.188 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1200 IOs, 560715776 bytes 2023-10-11T11:12:32.320 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1300 IOs, 588545024 bytes 2023-10-11T11:12:32.507 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1400 IOs, 622621696 bytes 2023-10-11T11:12:32.644 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1500 IOs, 640854016 bytes 2023-10-11T11:12:32.937 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1600 IOs, 738022400 bytes 2023-10-11T11:12:33.046 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1700 IOs, 739586048 bytes 2023-10-11T11:12:33.224 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1800 IOs, 776799232 bytes 2023-10-11T11:12:33.346 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1900 IOs, 782441472 bytes 2023-10-11T11:12:33.530 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2000 IOs, 819041280 bytes 2023-10-11T11:12:33.653 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2100 IOs, 824866816 bytes 2023-10-11T11:12:33.777 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2200 IOs, 835332096 bytes 2023-10-11T11:12:33.946 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2300 IOs, 881196032 bytes 2023-10-11T11:12:34.148 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2400 IOs, 917550080 bytes 2023-10-11T11:12:34.263 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2500 IOs, 925435904 bytes 2023-10-11T11:12:34.411 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2600 IOs, 956186624 bytes 2023-10-11T11:12:34.637 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2700 IOs, 1002093568 bytes 2023-10-11T11:12:34.885 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2800 IOs, 1068302336 bytes 2023-10-11T11:12:35.046 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2900 IOs, 1116593152 bytes 2023-10-11T11:12:35.283 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3000 IOs, 1166102528 bytes 2023-10-11T11:12:35.415 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3100 IOs, 1183041536 bytes 2023-10-11T11:12:35.591 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3200 IOs, 1213266944 bytes 2023-10-11T11:12:35.594 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.49845379185263916) > weight (0.25) 2023-10-11T11:12:35.595 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.7004723852530373) > weight (0.25) 2023-10-11T11:12:35.595 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.6631169276219359) > weight (0.25) 2023-10-11T11:12:35.595 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.8401989950479498) > weight (0.25) 2023-10-11T11:12:35.595 INFO:tasks.rbd_mirror_thrash:waiting for 81 secs before thrashing 2023-10-11T11:12:35.821 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3300 IOs, 1277294592 bytes 2023-10-11T11:12:36.006 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3400 IOs, 1314317312 bytes 2023-10-11T11:12:36.139 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3500 IOs, 1335556096 bytes 2023-10-11T11:12:36.312 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3600 IOs, 1374064640 bytes 2023-10-11T11:12:36.431 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3700 IOs, 1384435712 bytes 2023-10-11T11:12:36.629 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3800 IOs, 1441804288 bytes 2023-10-11T11:12:36.755 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3900 IOs, 1446844416 bytes 2023-10-11T11:12:36.955 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4000 IOs, 1509644288 bytes 2023-10-11T11:12:37.167 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4100 IOs, 1547030528 bytes 2023-10-11T11:12:37.297 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4200 IOs, 1563779072 bytes 2023-10-11T11:12:37.445 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4300 IOs, 1588888576 bytes 2023-10-11T11:12:37.617 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4400 IOs, 1641107456 bytes 2023-10-11T11:12:37.834 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4500 IOs, 1680381952 bytes 2023-10-11T11:12:37.989 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4600 IOs, 1708272640 bytes 2023-10-11T11:12:38.136 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4700 IOs, 1728897024 bytes 2023-10-11T11:12:38.320 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4800 IOs, 1772511232 bytes 2023-10-11T11:12:38.476 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4900 IOs, 1800407040 bytes 2023-10-11T11:12:38.873 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5000 IOs, 1863349248 bytes 2023-10-11T11:12:39.043 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5100 IOs, 1894190080 bytes 2023-10-11T11:12:39.212 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5200 IOs, 1943007232 bytes 2023-10-11T11:12:39.380 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5300 IOs, 1961982976 bytes 2023-10-11T11:12:39.488 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5400 IOs, 1974641664 bytes 2023-10-11T11:12:39.692 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5500 IOs, 2017794048 bytes 2023-10-11T11:12:39.852 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5600 IOs, 2046454784 bytes 2023-10-11T11:12:40.096 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5700 IOs, 2113862656 bytes 2023-10-11T11:12:40.231 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5800 IOs, 2120584192 bytes 2023-10-11T11:12:40.344 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5900 IOs, 2132645888 bytes 2023-10-11T11:12:40.530 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6000 IOs, 2183529472 bytes 2023-10-11T11:12:40.838 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6100 IOs, 2283373568 bytes 2023-10-11T11:12:41.010 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6200 IOs, 2312859648 bytes 2023-10-11T11:12:41.253 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6300 IOs, 2379558912 bytes 2023-10-11T11:12:41.415 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6400 IOs, 2412660736 bytes 2023-10-11T11:12:41.597 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6500 IOs, 2454768640 bytes 2023-10-11T11:12:41.755 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6600 IOs, 2484502528 bytes 2023-10-11T11:12:41.858 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6700 IOs, 2486523904 bytes 2023-10-11T11:12:41.970 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6800 IOs, 2495171584 bytes 2023-10-11T11:12:42.087 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6900 IOs, 2506094592 bytes 2023-10-11T11:12:42.267 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:7000 IOs, 2544997376 bytes 2023-10-11T11:12:42.498 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:7100 IOs, 2628327424 bytes 2023-10-11T11:12:42.666 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ error_code=124 2023-10-11T11:12:42.666 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ set -e 2023-10-11T11:12:42.666 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' 124 -eq 124 ']' 2023-10-11T11:12:42.667 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:12:42.667 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_present cluster1 mirror image_2 present 2023-10-11T11:12:42.667 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:12:42.667 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:12:42.667 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_2 2023-10-11T11:12:42.668 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local state=present 2023-10-11T11:12:42.668 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image_id= 2023-10-11T11:12:42.668 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T11:12:42.668 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test -n '' 2023-10-11T11:12:42.668 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ get_image_id cluster1 mirror image_2 2023-10-11T11:12:42.668 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:12:42.669 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local pool=mirror 2023-10-11T11:12:42.669 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local image=image_2 2023-10-11T11:12:42.669 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2023-10-11T11:12:42.669 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-10-11T11:12:42.710 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image_id=13166cd7570e 2023-10-11T11:12:42.711 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-10-11T11:12:42.711 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.1 2023-10-11T11:12:42.812 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test_image_present cluster1 mirror image_2 present 13166cd7570e 2023-10-11T11:12:42.813 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:12:42.813 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:12:42.813 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_2 2023-10-11T11:12:42.813 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local test_state=present 2023-10-11T11:12:42.813 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image_id=13166cd7570e 2023-10-11T11:12:42.813 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_state=deleted 2023-10-11T11:12:42.814 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_image_id 2023-10-11T11:12:42.814 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ get_image_id cluster1 mirror image_2 2023-10-11T11:12:42.814 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:12:42.814 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local pool=mirror 2023-10-11T11:12:42.814 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local image=image_2 2023-10-11T11:12:42.815 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2023-10-11T11:12:42.815 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-10-11T11:12:42.856 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ current_image_id=13166cd7570e 2023-10-11T11:12:42.856 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test -n 13166cd7570e 2023-10-11T11:12:42.856 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test -z 13166cd7570e -o 13166cd7570e = 13166cd7570e 2023-10-11T11:12:42.856 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ current_state=present 2023-10-11T11:12:42.856 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test present = present 2023-10-11T11:12:42.857 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:12:42.857 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ disable_mirror cluster2 mirror image_2 2023-10-11T11:12:42.857 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:12:42.857 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:12:42.857 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_2 2023-10-11T11:12:42.858 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_2 2023-10-11T11:12:55.759 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2023-10-11T11:12:55.760 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2023-10-11T11:12:55.760 INFO:tasks.rbd_mirror_thrash:waiting for 5 secs before reviving daemons 2023-10-11T11:12:55.761 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi111.stderr:2023-10-11T11:12:55.760+0000 7f23ff9be700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 131518) UID: 1000 2023-10-11T11:12:59.701 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi111.stderr:2023-10-11T11:12:59.700+0000 7f1e68fcb700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5569846251e0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-10-11T11:12:59.701 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi111.stderr:2023-10-11T11:12:59.700+0000 7f1e6efd7700 -1 rbd::mirror::ImageReplayer: 0x5569846e5900 [3/20741d1a-dcdc-4270-a299-73cd05081476] operator(): start failed: (85) Interrupted system call should be restarted 2023-10-11T11:13:00.761 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2023-10-11T11:13:00.761 INFO:teuthology.orchestra.run:waiting for 600 2023-10-11T11:13:00.762 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2023-10-11T11:13:00.762 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2023-10-11T11:13:00.762 DEBUG:teuthology.orchestra.run.smithi111:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2023-10-11T11:13:00.765 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2023-10-11T11:13:00.765 INFO:tasks.rbd_mirror_thrash:waiting for 111 secs before thrashing 2023-10-11T11:13:12.498 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi111.stderr:2023-10-11T11:13:12.497+0000 7f2c29ed3700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55c1e82de000 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=370 2023-10-11T11:13:12.498 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi111.stderr:2023-10-11T11:13:12.497+0000 7f2c29ed3700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55c1e82de000 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=374, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=9a182971-2e50-451f-a839-7c08d3b6157c, primary_snap_id=172, last_copied_object_number=128, snap_seqs={370=18446744073709551614}] 2023-10-11T11:13:12.510 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:Mirroring disabled 2023-10-11T11:13:12.525 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_present cluster1 mirror image_2 deleted 2023-10-11T11:13:12.525 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:13:12.525 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:13:12.526 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_2 2023-10-11T11:13:12.526 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local state=deleted 2023-10-11T11:13:12.526 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image_id= 2023-10-11T11:13:12.526 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T11:13:12.526 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test -n '' 2023-10-11T11:13:12.527 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ get_image_id cluster1 mirror image_2 2023-10-11T11:13:12.527 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:13:12.527 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local pool=mirror 2023-10-11T11:13:12.527 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local image=image_2 2023-10-11T11:13:12.527 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2023-10-11T11:13:12.528 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-10-11T11:13:12.562 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd: error opening image image_2: (2) No such file or directory 2023-10-11T11:13:12.565 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image_id= 2023-10-11T11:13:12.566 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-10-11T11:13:12.566 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.1 2023-10-11T11:13:12.667 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test_image_present cluster1 mirror image_2 deleted '' 2023-10-11T11:13:12.668 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:13:12.668 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:13:12.668 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_2 2023-10-11T11:13:12.668 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local test_state=deleted 2023-10-11T11:13:12.669 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image_id= 2023-10-11T11:13:12.669 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_state=deleted 2023-10-11T11:13:12.669 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_image_id 2023-10-11T11:13:12.669 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ get_image_id cluster1 mirror image_2 2023-10-11T11:13:12.669 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:13:12.670 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local pool=mirror 2023-10-11T11:13:12.670 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local image=image_2 2023-10-11T11:13:12.670 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2023-10-11T11:13:12.670 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-10-11T11:13:12.700 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd: error opening image image_2: (2) No such file or directory 2023-10-11T11:13:12.703 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ current_image_id= 2023-10-11T11:13:12.704 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test -n '' 2023-10-11T11:13:12.704 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test deleted = deleted 2023-10-11T11:13:12.704 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:13:12.704 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ purge_snapshots cluster2 mirror image_2 2023-10-11T11:13:12.705 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:13:12.705 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:13:12.705 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_2 2023-10-11T11:13:12.705 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 snap purge mirror/image_2 2023-10-11T11:13:12.747 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_image_retry cluster2 mirror image_2 2023-10-11T11:13:12.747 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:13:12.747 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:13:12.747 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_2 2023-10-11T11:13:12.748 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0 1 2 4 8 16 32 2023-10-11T11:13:12.748 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0 2023-10-11T11:13:12.748 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_image cluster2 mirror image_2 2023-10-11T11:13:12.749 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:13:12.749 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:13:12.749 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_2 2023-10-11T11:13:12.749 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_2 2023-10-11T11:13:12.790 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 rm mirror/image_2 2023-10-11T11:13:13.260 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2023-10-11T11:13:13.264 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:13:13.264 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 10` 2023-10-11T11:13:13.265 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_3 2023-10-11T11:13:13.265 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ create_image cluster2 mirror image_3 512M 2023-10-11T11:13:13.265 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:13:13.265 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T11:13:13.265 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:13:13.265 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T11:13:13.266 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_3 2023-10-11T11:13:13.266 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T11:13:13.266 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local size=128 2023-10-11T11:13:13.266 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n 512M ']' 2023-10-11T11:13:13.267 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ size=512M 2023-10-11T11:13:13.267 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T11:13:13.267 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_3 2023-10-11T11:13:13.313 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ enable_mirror cluster2 mirror image_3 2023-10-11T11:13:13.313 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:13:13.313 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:13:13.313 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_3 2023-10-11T11:13:13.313 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local mode=snapshot 2023-10-11T11:13:13.314 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_3 snapshot 2023-10-11T11:13:13.508 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:Mirroring enabled 2023-10-11T11:13:13.517 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 info mirror/image_3 2023-10-11T11:13:13.554 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:rbd image 'image_3': 2023-10-11T11:13:13.554 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: size 512 MiB in 128 objects 2023-10-11T11:13:13.554 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: order 22 (4 MiB objects) 2023-10-11T11:13:13.554 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: snapshot_count: 1 2023-10-11T11:13:13.554 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: id: 18e1f42a5cab 2023-10-11T11:13:13.555 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: block_name_prefix: rbd_data.18e1f42a5cab 2023-10-11T11:13:13.555 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: format: 2 2023-10-11T11:13:13.555 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-10-11T11:13:13.555 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: op_features: 2023-10-11T11:13:13.555 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: flags: 2023-10-11T11:13:13.555 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: create_timestamp: Wed Oct 11 11:13:13 2023 2023-10-11T11:13:13.556 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: access_timestamp: Wed Oct 11 11:13:13 2023 2023-10-11T11:13:13.556 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: modify_timestamp: Wed Oct 11 11:13:13 2023 2023-10-11T11:13:13.556 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring state: enabled 2023-10-11T11:13:13.556 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring mode: snapshot 2023-10-11T11:13:13.556 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring global id: 4025e229-0f14-4955-a2b2-2cd12c42f74d 2023-10-11T11:13:13.557 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring primary: true 2023-10-11T11:13:13.557 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ stress_write_image cluster2 mirror image_3 2023-10-11T11:13:13.557 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:13:13.558 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:13:13.558 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_3 2023-10-11T11:13:13.558 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-10-11T11:13:13.559 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local duration=14 2023-10-11T11:13:13.559 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ set +e 2023-10-11T11:13:13.559 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_3 --debug-rbd=20 --debug-journaler=20 2023-10-11T11:13:13.870 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:100 IOs, 47333376 bytes 2023-10-11T11:13:14.102 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:200 IOs, 96757760 bytes 2023-10-11T11:13:14.300 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:300 IOs, 141915136 bytes 2023-10-11T11:13:14.472 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:400 IOs, 173879296 bytes 2023-10-11T11:13:14.700 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:500 IOs, 233663488 bytes 2023-10-11T11:13:14.839 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:600 IOs, 255809536 bytes 2023-10-11T11:13:14.944 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:700 IOs, 256473088 bytes 2023-10-11T11:13:15.189 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:800 IOs, 327456768 bytes 2023-10-11T11:13:15.356 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:900 IOs, 363426816 bytes 2023-10-11T11:13:15.524 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1000 IOs, 401285120 bytes 2023-10-11T11:13:15.639 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1100 IOs, 411516928 bytes 2023-10-11T11:13:15.862 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1200 IOs, 467785728 bytes 2023-10-11T11:13:16.124 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1300 IOs, 551746560 bytes 2023-10-11T11:13:16.389 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1400 IOs, 597778432 bytes 2023-10-11T11:13:16.575 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1500 IOs, 608074752 bytes 2023-10-11T11:13:16.810 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1600 IOs, 668560384 bytes 2023-10-11T11:13:17.018 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1700 IOs, 720741376 bytes 2023-10-11T11:13:17.316 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1800 IOs, 839982080 bytes 2023-10-11T11:13:17.564 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1900 IOs, 883251200 bytes 2023-10-11T11:13:17.754 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2000 IOs, 925201408 bytes 2023-10-11T11:13:17.928 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2100 IOs, 970293248 bytes 2023-10-11T11:13:18.141 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2200 IOs, 1007730688 bytes 2023-10-11T11:13:18.298 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2300 IOs, 1042872320 bytes 2023-10-11T11:13:18.547 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2400 IOs, 1106920448 bytes 2023-10-11T11:13:18.765 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2500 IOs, 1177601024 bytes 2023-10-11T11:13:18.898 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2600 IOs, 1180647424 bytes 2023-10-11T11:13:19.034 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2700 IOs, 1197929472 bytes 2023-10-11T11:13:19.205 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2800 IOs, 1231395840 bytes 2023-10-11T11:13:19.392 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2900 IOs, 1276435456 bytes 2023-10-11T11:13:19.513 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3000 IOs, 1287240704 bytes 2023-10-11T11:13:19.761 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3100 IOs, 1360830464 bytes 2023-10-11T11:13:19.957 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3200 IOs, 1400158208 bytes 2023-10-11T11:13:20.170 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3300 IOs, 1445340160 bytes 2023-10-11T11:13:20.298 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3400 IOs, 1467661312 bytes 2023-10-11T11:13:20.495 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3500 IOs, 1490795520 bytes 2023-10-11T11:13:20.678 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3600 IOs, 1530508288 bytes 2023-10-11T11:13:20.796 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3700 IOs, 1534946304 bytes 2023-10-11T11:13:20.988 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3800 IOs, 1582341120 bytes 2023-10-11T11:13:21.172 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3900 IOs, 1625568256 bytes 2023-10-11T11:13:21.325 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4000 IOs, 1648912384 bytes 2023-10-11T11:13:21.423 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4100 IOs, 1649536000 bytes 2023-10-11T11:13:21.561 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4200 IOs, 1666887680 bytes 2023-10-11T11:13:21.686 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4300 IOs, 1682537472 bytes 2023-10-11T11:13:21.890 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4400 IOs, 1729864704 bytes 2023-10-11T11:13:22.037 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4500 IOs, 1776925696 bytes 2023-10-11T11:13:22.216 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4600 IOs, 1813836800 bytes 2023-10-11T11:13:22.416 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4700 IOs, 1845372928 bytes 2023-10-11T11:13:22.650 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4800 IOs, 1913050112 bytes 2023-10-11T11:13:22.807 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4900 IOs, 1943363584 bytes 2023-10-11T11:13:22.985 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5000 IOs, 1984055296 bytes 2023-10-11T11:13:23.261 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5100 IOs, 2079627264 bytes 2023-10-11T11:13:23.382 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5200 IOs, 2088528896 bytes 2023-10-11T11:13:23.566 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5300 IOs, 2124566528 bytes 2023-10-11T11:13:23.732 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5400 IOs, 2155796480 bytes 2023-10-11T11:13:23.894 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5500 IOs, 2192143360 bytes 2023-10-11T11:13:24.156 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5600 IOs, 2263329792 bytes 2023-10-11T11:13:24.326 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5700 IOs, 2300200960 bytes 2023-10-11T11:13:24.462 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5800 IOs, 2320909312 bytes 2023-10-11T11:13:24.604 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5900 IOs, 2339929088 bytes 2023-10-11T11:13:24.793 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6000 IOs, 2374859776 bytes 2023-10-11T11:13:25.028 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6100 IOs, 2446518272 bytes 2023-10-11T11:13:25.204 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6200 IOs, 2487114752 bytes 2023-10-11T11:13:25.416 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6300 IOs, 2545138688 bytes 2023-10-11T11:13:25.598 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6400 IOs, 2587318272 bytes 2023-10-11T11:13:25.757 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6500 IOs, 2617550848 bytes 2023-10-11T11:13:26.015 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6600 IOs, 2695265280 bytes 2023-10-11T11:13:26.241 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6700 IOs, 2749851648 bytes 2023-10-11T11:13:26.529 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6800 IOs, 2852883456 bytes 2023-10-11T11:13:26.850 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6900 IOs, 2957402112 bytes 2023-10-11T11:13:27.068 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:7000 IOs, 3018752000 bytes 2023-10-11T11:13:27.230 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:7100 IOs, 3051809792 bytes 2023-10-11T11:13:27.421 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:7200 IOs, 3097624576 bytes 2023-10-11T11:13:27.567 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ error_code=124 2023-10-11T11:13:27.567 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ set -e 2023-10-11T11:13:27.567 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' 124 -eq 124 ']' 2023-10-11T11:13:27.568 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:13:27.568 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_present cluster1 mirror image_3 present 2023-10-11T11:13:27.568 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:13:27.568 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:13:27.568 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_3 2023-10-11T11:13:27.569 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local state=present 2023-10-11T11:13:27.569 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image_id= 2023-10-11T11:13:27.569 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T11:13:27.569 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test -n '' 2023-10-11T11:13:27.570 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ get_image_id cluster1 mirror image_3 2023-10-11T11:13:27.570 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:13:27.570 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local pool=mirror 2023-10-11T11:13:27.570 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local image=image_3 2023-10-11T11:13:27.570 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2023-10-11T11:13:27.571 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-10-11T11:13:27.609 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image_id=13169b7ce34f 2023-10-11T11:13:27.609 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-10-11T11:13:27.609 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.1 2023-10-11T11:13:27.711 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test_image_present cluster1 mirror image_3 present 13169b7ce34f 2023-10-11T11:13:27.711 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:13:27.711 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:13:27.711 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_3 2023-10-11T11:13:27.711 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local test_state=present 2023-10-11T11:13:27.712 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image_id=13169b7ce34f 2023-10-11T11:13:27.712 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_state=deleted 2023-10-11T11:13:27.712 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_image_id 2023-10-11T11:13:27.712 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ get_image_id cluster1 mirror image_3 2023-10-11T11:13:27.712 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:13:27.712 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local pool=mirror 2023-10-11T11:13:27.713 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local image=image_3 2023-10-11T11:13:27.713 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2023-10-11T11:13:27.713 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-10-11T11:13:27.754 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ current_image_id=13169b7ce34f 2023-10-11T11:13:27.754 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test -n 13169b7ce34f 2023-10-11T11:13:27.755 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test -z 13169b7ce34f -o 13169b7ce34f = 13169b7ce34f 2023-10-11T11:13:27.755 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ current_state=present 2023-10-11T11:13:27.755 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test present = present 2023-10-11T11:13:27.755 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:13:27.755 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ disable_mirror cluster2 mirror image_3 2023-10-11T11:13:27.756 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:13:27.756 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:13:27.756 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_3 2023-10-11T11:13:27.756 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_3 2023-10-11T11:13:29.701 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi111.stderr:2023-10-11T11:13:29.700+0000 7f1e68fcb700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5569849f8ea0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-10-11T11:13:29.701 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi111.stderr:2023-10-11T11:13:29.700+0000 7f1e6efd7700 -1 rbd::mirror::ImageReplayer: 0x5569846e5900 [3/4025e229-0f14-4955-a2b2-2cd12c42f74d] operator(): start failed: (85) Interrupted system call should be restarted 2023-10-11T11:13:56.596 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2023-10-11T11:13:56.597 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2023-10-11T11:13:56.597 INFO:tasks.rbd_mirror_thrash:waiting for 1 secs before reviving daemons 2023-10-11T11:13:56.598 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi111.stderr:2023-10-11T11:13:56.597+0000 7fbdf15c3700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 125066) UID: 1000 2023-10-11T11:13:57.598 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2023-10-11T11:13:57.598 INFO:teuthology.orchestra.run:waiting for 600 2023-10-11T11:13:57.599 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2023-10-11T11:13:57.599 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2023-10-11T11:13:57.599 DEBUG:teuthology.orchestra.run.smithi111:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2023-10-11T11:13:57.603 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2023-10-11T11:13:57.603 INFO:tasks.rbd_mirror_thrash:waiting for 110 secs before thrashing 2023-10-11T11:13:57.627 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi111.stderr:2023-10-11T11:13:57.626+0000 7f2c296d2700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55c1e972ac00 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=372 2023-10-11T11:13:57.628 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi111.stderr:2023-10-11T11:13:57.626+0000 7f2c296d2700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55c1e972ac00 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=376, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=9a182971-2e50-451f-a839-7c08d3b6157c, primary_snap_id=174, last_copied_object_number=128, snap_seqs={372=18446744073709551614}] 2023-10-11T11:13:57.633 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:Mirroring disabled 2023-10-11T11:13:57.649 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_present cluster1 mirror image_3 deleted 2023-10-11T11:13:57.649 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:13:57.649 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:13:57.650 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_3 2023-10-11T11:13:57.650 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local state=deleted 2023-10-11T11:13:57.650 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image_id= 2023-10-11T11:13:57.650 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T11:13:57.651 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test -n '' 2023-10-11T11:13:57.651 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ get_image_id cluster1 mirror image_3 2023-10-11T11:13:57.651 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:13:57.651 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local pool=mirror 2023-10-11T11:13:57.651 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local image=image_3 2023-10-11T11:13:57.652 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2023-10-11T11:13:57.652 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-10-11T11:13:57.686 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd: error opening image image_3: (2) No such file or directory 2023-10-11T11:13:57.690 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image_id= 2023-10-11T11:13:57.690 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-10-11T11:13:57.690 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.1 2023-10-11T11:13:57.792 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test_image_present cluster1 mirror image_3 deleted '' 2023-10-11T11:13:57.792 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:13:57.792 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:13:57.793 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_3 2023-10-11T11:13:57.793 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local test_state=deleted 2023-10-11T11:13:57.793 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image_id= 2023-10-11T11:13:57.793 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_state=deleted 2023-10-11T11:13:57.793 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_image_id 2023-10-11T11:13:57.794 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ get_image_id cluster1 mirror image_3 2023-10-11T11:13:57.794 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:13:57.794 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local pool=mirror 2023-10-11T11:13:57.794 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local image=image_3 2023-10-11T11:13:57.794 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2023-10-11T11:13:57.795 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-10-11T11:13:57.828 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd: error opening image image_3: (2) No such file or directory 2023-10-11T11:13:57.832 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ current_image_id= 2023-10-11T11:13:57.832 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test -n '' 2023-10-11T11:13:57.833 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test deleted = deleted 2023-10-11T11:13:57.833 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:13:57.833 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ purge_snapshots cluster2 mirror image_3 2023-10-11T11:13:57.833 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:13:57.833 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:13:57.834 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_3 2023-10-11T11:13:57.834 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 snap purge mirror/image_3 2023-10-11T11:13:57.877 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_image_retry cluster2 mirror image_3 2023-10-11T11:13:57.877 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:13:57.877 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:13:57.877 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_3 2023-10-11T11:13:57.878 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0 1 2 4 8 16 32 2023-10-11T11:13:57.878 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0 2023-10-11T11:13:57.879 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_image cluster2 mirror image_3 2023-10-11T11:13:57.879 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:13:57.879 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:13:57.879 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_3 2023-10-11T11:13:57.880 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_3 2023-10-11T11:13:57.920 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 rm mirror/image_3 2023-10-11T11:13:58.200 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2023-10-11T11:13:58.204 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:13:58.204 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 10` 2023-10-11T11:13:58.205 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_4 2023-10-11T11:13:58.205 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ create_image cluster2 mirror image_4 512M 2023-10-11T11:13:58.205 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:13:58.205 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T11:13:58.206 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:13:58.206 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T11:13:58.206 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_4 2023-10-11T11:13:58.206 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T11:13:58.207 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local size=128 2023-10-11T11:13:58.207 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n 512M ']' 2023-10-11T11:13:58.207 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ size=512M 2023-10-11T11:13:58.207 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T11:13:58.208 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_4 2023-10-11T11:13:58.253 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ enable_mirror cluster2 mirror image_4 2023-10-11T11:13:58.253 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:13:58.253 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:13:58.254 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_4 2023-10-11T11:13:58.254 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local mode=snapshot 2023-10-11T11:13:58.254 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_4 snapshot 2023-10-11T11:13:58.634 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:Mirroring enabled 2023-10-11T11:13:58.645 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 info mirror/image_4 2023-10-11T11:13:58.681 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:rbd image 'image_4': 2023-10-11T11:13:58.681 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: size 512 MiB in 128 objects 2023-10-11T11:13:58.681 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: order 22 (4 MiB objects) 2023-10-11T11:13:58.682 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: snapshot_count: 1 2023-10-11T11:13:58.682 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: id: 19025b70b49c 2023-10-11T11:13:58.682 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: block_name_prefix: rbd_data.19025b70b49c 2023-10-11T11:13:58.682 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: format: 2 2023-10-11T11:13:58.683 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-10-11T11:13:58.683 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: op_features: 2023-10-11T11:13:58.683 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: flags: 2023-10-11T11:13:58.684 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: create_timestamp: Wed Oct 11 11:13:58 2023 2023-10-11T11:13:58.684 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: access_timestamp: Wed Oct 11 11:13:58 2023 2023-10-11T11:13:58.684 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: modify_timestamp: Wed Oct 11 11:13:58 2023 2023-10-11T11:13:58.685 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring state: enabled 2023-10-11T11:13:58.685 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring mode: snapshot 2023-10-11T11:13:58.685 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring global id: 02f9ed79-be48-4b1c-9cbb-af23a4454e3f 2023-10-11T11:13:58.686 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring primary: true 2023-10-11T11:13:58.687 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ stress_write_image cluster2 mirror image_4 2023-10-11T11:13:58.687 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:13:58.688 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:13:58.688 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_4 2023-10-11T11:13:58.688 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-10-11T11:13:58.688 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local duration=14 2023-10-11T11:13:58.689 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ set +e 2023-10-11T11:13:58.689 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_4 --debug-rbd=20 --debug-journaler=20 2023-10-11T11:13:58.976 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:100 IOs, 58587136 bytes 2023-10-11T11:13:59.204 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:200 IOs, 84806656 bytes 2023-10-11T11:13:59.434 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:300 IOs, 144572416 bytes 2023-10-11T11:13:59.587 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:400 IOs, 196798464 bytes 2023-10-11T11:13:59.891 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:500 IOs, 267264000 bytes 2023-10-11T11:14:00.098 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:600 IOs, 315894784 bytes 2023-10-11T11:14:00.313 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:700 IOs, 386340864 bytes 2023-10-11T11:14:00.614 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:800 IOs, 469627904 bytes 2023-10-11T11:14:00.794 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:900 IOs, 501277696 bytes 2023-10-11T11:14:00.940 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1000 IOs, 522797056 bytes 2023-10-11T11:14:01.100 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1100 IOs, 553947136 bytes 2023-10-11T11:14:01.223 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1200 IOs, 563525632 bytes 2023-10-11T11:14:01.391 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1300 IOs, 598535168 bytes 2023-10-11T11:14:01.600 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1400 IOs, 643632128 bytes 2023-10-11T11:14:01.829 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1500 IOs, 695719936 bytes 2023-10-11T11:14:01.977 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1600 IOs, 721650688 bytes 2023-10-11T11:14:02.181 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1700 IOs, 791833600 bytes 2023-10-11T11:14:02.371 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1800 IOs, 814617600 bytes 2023-10-11T11:14:02.516 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1900 IOs, 831566848 bytes 2023-10-11T11:14:02.741 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2000 IOs, 891621376 bytes 2023-10-11T11:14:02.854 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2100 IOs, 892247040 bytes 2023-10-11T11:14:03.092 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2200 IOs, 987298816 bytes 2023-10-11T11:14:03.284 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2300 IOs, 1006081024 bytes 2023-10-11T11:14:03.460 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2400 IOs, 1042512896 bytes 2023-10-11T11:14:03.656 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2500 IOs, 1094779904 bytes 2023-10-11T11:14:03.928 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2600 IOs, 1188659200 bytes 2023-10-11T11:14:04.206 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2700 IOs, 1263946752 bytes 2023-10-11T11:14:04.320 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2800 IOs, 1264574464 bytes 2023-10-11T11:14:04.544 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2900 IOs, 1318245376 bytes 2023-10-11T11:14:04.713 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3000 IOs, 1346912256 bytes 2023-10-11T11:14:04.982 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3100 IOs, 1432768512 bytes 2023-10-11T11:14:05.104 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3200 IOs, 1449098240 bytes 2023-10-11T11:14:05.360 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3300 IOs, 1522138112 bytes 2023-10-11T11:14:05.565 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3400 IOs, 1560084480 bytes 2023-10-11T11:14:05.704 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3500 IOs, 1581252608 bytes 2023-10-11T11:14:05.910 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3600 IOs, 1632288768 bytes 2023-10-11T11:14:06.116 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3700 IOs, 1677883392 bytes 2023-10-11T11:14:06.355 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3800 IOs, 1764639744 bytes 2023-10-11T11:14:06.641 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3900 IOs, 1827222528 bytes 2023-10-11T11:14:06.807 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4000 IOs, 1862183936 bytes 2023-10-11T11:14:06.921 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4100 IOs, 1883166720 bytes 2023-10-11T11:14:07.114 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4200 IOs, 1914002432 bytes 2023-10-11T11:14:07.316 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4300 IOs, 1960092672 bytes 2023-10-11T11:14:07.425 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4400 IOs, 1960715264 bytes 2023-10-11T11:14:07.596 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4500 IOs, 1993555968 bytes 2023-10-11T11:14:07.749 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4600 IOs, 2013300736 bytes 2023-10-11T11:14:07.924 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4700 IOs, 2045686784 bytes 2023-10-11T11:14:08.153 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4800 IOs, 2110504960 bytes 2023-10-11T11:14:08.290 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4900 IOs, 2127221760 bytes 2023-10-11T11:14:08.396 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5000 IOs, 2127838208 bytes 2023-10-11T11:14:08.617 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5100 IOs, 2187821056 bytes 2023-10-11T11:14:08.867 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5200 IOs, 2261134336 bytes 2023-10-11T11:14:09.087 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5300 IOs, 2315443200 bytes 2023-10-11T11:14:09.383 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5400 IOs, 2414510080 bytes 2023-10-11T11:14:09.575 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5500 IOs, 2461895680 bytes 2023-10-11T11:14:09.688 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5600 IOs, 2491000832 bytes 2023-10-11T11:14:09.797 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5700 IOs, 2491622400 bytes 2023-10-11T11:14:09.918 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5800 IOs, 2499232768 bytes 2023-10-11T11:14:10.162 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5900 IOs, 2569693184 bytes 2023-10-11T11:14:10.295 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6000 IOs, 2582871040 bytes 2023-10-11T11:14:10.487 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6100 IOs, 2624524288 bytes 2023-10-11T11:14:10.663 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6200 IOs, 2660663296 bytes 2023-10-11T11:14:10.863 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6300 IOs, 2706765824 bytes 2023-10-11T11:14:11.008 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6400 IOs, 2721195008 bytes 2023-10-11T11:14:11.164 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6500 IOs, 2747487232 bytes 2023-10-11T11:14:11.391 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6600 IOs, 2809692160 bytes 2023-10-11T11:14:11.595 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6700 IOs, 2858455040 bytes 2023-10-11T11:14:11.879 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6800 IOs, 2947768320 bytes 2023-10-11T11:14:12.124 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6900 IOs, 3037820928 bytes 2023-10-11T11:14:12.338 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:7000 IOs, 3075570688 bytes 2023-10-11T11:14:12.532 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:7100 IOs, 3114126336 bytes 2023-10-11T11:14:12.691 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ error_code=124 2023-10-11T11:14:12.691 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ set -e 2023-10-11T11:14:12.692 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' 124 -eq 124 ']' 2023-10-11T11:14:12.692 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:14:12.692 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_present cluster1 mirror image_4 present 2023-10-11T11:14:12.692 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:14:12.693 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:14:12.693 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_4 2023-10-11T11:14:12.693 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local state=present 2023-10-11T11:14:12.694 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image_id= 2023-10-11T11:14:12.694 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T11:14:12.694 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test -n '' 2023-10-11T11:14:12.694 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ get_image_id cluster1 mirror image_4 2023-10-11T11:14:12.695 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:14:12.695 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local pool=mirror 2023-10-11T11:14:12.695 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local image=image_4 2023-10-11T11:14:12.695 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2023-10-11T11:14:12.696 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-10-11T11:14:12.733 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image_id=1316a6010b33 2023-10-11T11:14:12.734 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-10-11T11:14:12.734 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.1 2023-10-11T11:14:12.835 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test_image_present cluster1 mirror image_4 present 1316a6010b33 2023-10-11T11:14:12.835 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:14:12.835 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:14:12.835 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_4 2023-10-11T11:14:12.836 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local test_state=present 2023-10-11T11:14:12.836 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image_id=1316a6010b33 2023-10-11T11:14:12.836 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_state=deleted 2023-10-11T11:14:12.836 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_image_id 2023-10-11T11:14:12.837 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ get_image_id cluster1 mirror image_4 2023-10-11T11:14:12.837 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:14:12.837 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local pool=mirror 2023-10-11T11:14:12.837 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local image=image_4 2023-10-11T11:14:12.837 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2023-10-11T11:14:12.838 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-10-11T11:14:12.878 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ current_image_id=1316a6010b33 2023-10-11T11:14:12.878 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test -n 1316a6010b33 2023-10-11T11:14:12.878 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test -z 1316a6010b33 -o 1316a6010b33 = 1316a6010b33 2023-10-11T11:14:12.878 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ current_state=present 2023-10-11T11:14:12.879 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test present = present 2023-10-11T11:14:12.879 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:14:12.879 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ disable_mirror cluster2 mirror image_4 2023-10-11T11:14:12.879 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:14:12.879 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:14:12.880 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_4 2023-10-11T11:14:12.880 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_4 2023-10-11T11:14:29.702 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi111.stderr:2023-10-11T11:14:29.701+0000 7f1e687ca700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5569849f96c0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-10-11T11:14:29.702 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi111.stderr:2023-10-11T11:14:29.701+0000 7f1e6efd7700 -1 rbd::mirror::ImageReplayer: 0x5569846e5900 [3/02f9ed79-be48-4b1c-9cbb-af23a4454e3f] operator(): start failed: (85) Interrupted system call should be restarted 2023-10-11T11:14:43.541 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi111.stderr:2023-10-11T11:14:43.539+0000 7f2c296d2700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55c1e751e800 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=374 2023-10-11T11:14:43.541 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi111.stderr:2023-10-11T11:14:43.539+0000 7f2c296d2700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55c1e751e800 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=378, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=9a182971-2e50-451f-a839-7c08d3b6157c, primary_snap_id=176, last_copied_object_number=128, snap_seqs={374=18446744073709551614}] 2023-10-11T11:14:43.552 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:Mirroring disabled 2023-10-11T11:14:43.568 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_present cluster1 mirror image_4 deleted 2023-10-11T11:14:43.568 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:14:43.568 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:14:43.569 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_4 2023-10-11T11:14:43.569 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local state=deleted 2023-10-11T11:14:43.569 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image_id= 2023-10-11T11:14:43.569 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T11:14:43.569 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test -n '' 2023-10-11T11:14:43.570 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ get_image_id cluster1 mirror image_4 2023-10-11T11:14:43.570 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:14:43.570 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local pool=mirror 2023-10-11T11:14:43.570 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local image=image_4 2023-10-11T11:14:43.571 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2023-10-11T11:14:43.571 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-10-11T11:14:43.605 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd: error opening image image_4: (2) No such file or directory 2023-10-11T11:14:43.609 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image_id= 2023-10-11T11:14:43.609 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-10-11T11:14:43.609 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.1 2023-10-11T11:14:43.711 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test_image_present cluster1 mirror image_4 deleted '' 2023-10-11T11:14:43.711 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:14:43.712 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:14:43.712 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_4 2023-10-11T11:14:43.712 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local test_state=deleted 2023-10-11T11:14:43.712 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image_id= 2023-10-11T11:14:43.712 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_state=deleted 2023-10-11T11:14:43.713 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_image_id 2023-10-11T11:14:43.713 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ get_image_id cluster1 mirror image_4 2023-10-11T11:14:43.713 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:14:43.713 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local pool=mirror 2023-10-11T11:14:43.713 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local image=image_4 2023-10-11T11:14:43.714 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2023-10-11T11:14:43.714 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-10-11T11:14:43.744 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd: error opening image image_4: (2) No such file or directory 2023-10-11T11:14:43.747 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ current_image_id= 2023-10-11T11:14:43.748 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test -n '' 2023-10-11T11:14:43.748 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test deleted = deleted 2023-10-11T11:14:43.748 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:14:43.748 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ purge_snapshots cluster2 mirror image_4 2023-10-11T11:14:43.748 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:14:43.749 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:14:43.749 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_4 2023-10-11T11:14:43.749 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 snap purge mirror/image_4 2023-10-11T11:14:43.790 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_image_retry cluster2 mirror image_4 2023-10-11T11:14:43.791 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:14:43.791 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:14:43.791 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_4 2023-10-11T11:14:43.791 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0 1 2 4 8 16 32 2023-10-11T11:14:43.792 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0 2023-10-11T11:14:43.792 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_image cluster2 mirror image_4 2023-10-11T11:14:43.792 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:14:43.793 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:14:43.793 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_4 2023-10-11T11:14:43.793 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_4 2023-10-11T11:14:43.834 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 rm mirror/image_4 2023-10-11T11:14:44.112 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2023-10-11T11:14:44.116 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:14:44.116 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 10` 2023-10-11T11:14:44.116 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_5 2023-10-11T11:14:44.116 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ create_image cluster2 mirror image_5 512M 2023-10-11T11:14:44.117 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:14:44.117 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T11:14:44.117 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:14:44.117 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T11:14:44.118 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_5 2023-10-11T11:14:44.118 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T11:14:44.118 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local size=128 2023-10-11T11:14:44.118 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n 512M ']' 2023-10-11T11:14:44.118 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ size=512M 2023-10-11T11:14:44.119 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T11:14:44.119 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_5 2023-10-11T11:14:44.164 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ enable_mirror cluster2 mirror image_5 2023-10-11T11:14:44.164 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:14:44.165 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:14:44.165 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_5 2023-10-11T11:14:44.165 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local mode=snapshot 2023-10-11T11:14:44.165 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_5 snapshot 2023-10-11T11:14:44.546 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:Mirroring enabled 2023-10-11T11:14:44.557 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 info mirror/image_5 2023-10-11T11:14:44.592 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:rbd image 'image_5': 2023-10-11T11:14:44.592 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: size 512 MiB in 128 objects 2023-10-11T11:14:44.592 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: order 22 (4 MiB objects) 2023-10-11T11:14:44.593 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: snapshot_count: 1 2023-10-11T11:14:44.593 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: id: 191a4e3703c4 2023-10-11T11:14:44.593 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: block_name_prefix: rbd_data.191a4e3703c4 2023-10-11T11:14:44.593 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: format: 2 2023-10-11T11:14:44.594 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-10-11T11:14:44.594 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: op_features: 2023-10-11T11:14:44.594 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: flags: 2023-10-11T11:14:44.594 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: create_timestamp: Wed Oct 11 11:14:44 2023 2023-10-11T11:14:44.595 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: access_timestamp: Wed Oct 11 11:14:44 2023 2023-10-11T11:14:44.595 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: modify_timestamp: Wed Oct 11 11:14:44 2023 2023-10-11T11:14:44.595 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring state: enabled 2023-10-11T11:14:44.595 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring mode: snapshot 2023-10-11T11:14:44.596 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring global id: ed1be0f3-19ac-46e3-a670-bd5d93bbe097 2023-10-11T11:14:44.596 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring primary: true 2023-10-11T11:14:44.597 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ stress_write_image cluster2 mirror image_5 2023-10-11T11:14:44.598 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:14:44.598 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:14:44.598 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_5 2023-10-11T11:14:44.598 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-10-11T11:14:44.599 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local duration=12 2023-10-11T11:14:44.599 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ set +e 2023-10-11T11:14:44.599 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_5 --debug-rbd=20 --debug-journaler=20 2023-10-11T11:14:44.836 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:100 IOs, 4352000 bytes 2023-10-11T11:14:45.088 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:200 IOs, 66243584 bytes 2023-10-11T11:14:45.346 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:300 IOs, 145822720 bytes 2023-10-11T11:14:45.448 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:400 IOs, 150766592 bytes 2023-10-11T11:14:45.671 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:500 IOs, 205128704 bytes 2023-10-11T11:14:45.933 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:600 IOs, 290450432 bytes 2023-10-11T11:14:46.154 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:700 IOs, 361794560 bytes 2023-10-11T11:14:46.306 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:800 IOs, 383697920 bytes 2023-10-11T11:14:46.521 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:900 IOs, 454125568 bytes 2023-10-11T11:14:46.802 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1000 IOs, 532367360 bytes 2023-10-11T11:14:47.060 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1100 IOs, 604877824 bytes 2023-10-11T11:14:47.210 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1200 IOs, 627116032 bytes 2023-10-11T11:14:47.531 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1300 IOs, 732905472 bytes 2023-10-11T11:14:47.714 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1400 IOs, 771917824 bytes 2023-10-11T11:14:47.850 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1500 IOs, 789170176 bytes 2023-10-11T11:14:48.070 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1600 IOs, 852792320 bytes 2023-10-11T11:14:48.180 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1700 IOs, 853417984 bytes 2023-10-11T11:14:48.442 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1800 IOs, 894650368 bytes 2023-10-11T11:14:48.627 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1900 IOs, 927272960 bytes 2023-10-11T11:14:48.735 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2000 IOs, 931968000 bytes 2023-10-11T11:14:49.008 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2100 IOs, 932589568 bytes 2023-10-11T11:14:49.129 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2200 IOs, 942555136 bytes 2023-10-11T11:14:49.356 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2300 IOs, 1008694272 bytes 2023-10-11T11:14:49.582 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2400 IOs, 1063204864 bytes 2023-10-11T11:14:49.782 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2500 IOs, 1108984832 bytes 2023-10-11T11:14:50.217 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2600 IOs, 1174215680 bytes 2023-10-11T11:14:50.435 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2700 IOs, 1232208896 bytes 2023-10-11T11:14:50.642 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2800 IOs, 1295628288 bytes 2023-10-11T11:14:50.808 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2900 IOs, 1299909632 bytes 2023-10-11T11:14:50.997 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3000 IOs, 1342439424 bytes 2023-10-11T11:14:51.238 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3100 IOs, 1407028224 bytes 2023-10-11T11:14:51.351 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3200 IOs, 1409914880 bytes 2023-10-11T11:14:51.534 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3300 IOs, 1453952000 bytes 2023-10-11T11:14:51.644 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3400 IOs, 1454553088 bytes 2023-10-11T11:14:51.766 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.8634599138615277) > weight (0.25) 2023-10-11T11:14:51.766 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.5028776610705459) > weight (0.25) 2023-10-11T11:14:51.766 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.5213696213647889) > weight (0.25) 2023-10-11T11:14:51.767 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.8222011361076331) > weight (0.25) 2023-10-11T11:14:51.767 INFO:tasks.rbd_mirror_thrash:waiting for 106 secs before thrashing 2023-10-11T11:14:51.828 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3500 IOs, 1504873472 bytes 2023-10-11T11:14:52.059 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3600 IOs, 1553240064 bytes 2023-10-11T11:14:52.237 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3700 IOs, 1591057408 bytes 2023-10-11T11:14:52.381 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3800 IOs, 1609485312 bytes 2023-10-11T11:14:52.501 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3900 IOs, 1640488960 bytes 2023-10-11T11:14:52.857 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4000 IOs, 1744070656 bytes 2023-10-11T11:14:53.091 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4100 IOs, 1811877888 bytes 2023-10-11T11:14:53.227 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4200 IOs, 1828712448 bytes 2023-10-11T11:14:53.360 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4300 IOs, 1844345856 bytes 2023-10-11T11:14:53.633 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4400 IOs, 1920630784 bytes 2023-10-11T11:14:53.767 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4500 IOs, 1937895424 bytes 2023-10-11T11:14:53.914 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4600 IOs, 1959343104 bytes 2023-10-11T11:14:54.096 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4700 IOs, 2000584704 bytes 2023-10-11T11:14:54.253 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4800 IOs, 2024541184 bytes 2023-10-11T11:14:54.366 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4900 IOs, 2025168896 bytes 2023-10-11T11:14:54.587 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5000 IOs, 2089487360 bytes 2023-10-11T11:14:54.823 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5100 IOs, 2155736064 bytes 2023-10-11T11:14:54.988 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5200 IOs, 2184410112 bytes 2023-10-11T11:14:55.239 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5300 IOs, 2251058176 bytes 2023-10-11T11:14:55.449 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5400 IOs, 2308521984 bytes 2023-10-11T11:14:55.692 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5500 IOs, 2381109248 bytes 2023-10-11T11:14:55.854 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5600 IOs, 2413806592 bytes 2023-10-11T11:14:56.072 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5700 IOs, 2471822336 bytes 2023-10-11T11:14:56.228 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5800 IOs, 2496331776 bytes 2023-10-11T11:14:56.335 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5900 IOs, 2499275776 bytes 2023-10-11T11:14:56.528 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6000 IOs, 2564190208 bytes 2023-10-11T11:14:56.602 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ error_code=124 2023-10-11T11:14:56.603 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ set -e 2023-10-11T11:14:56.603 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' 124 -eq 124 ']' 2023-10-11T11:14:56.603 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:14:56.603 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_present cluster1 mirror image_5 present 2023-10-11T11:14:56.603 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:14:56.604 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:14:56.604 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_5 2023-10-11T11:14:56.604 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local state=present 2023-10-11T11:14:56.604 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image_id= 2023-10-11T11:14:56.604 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T11:14:56.604 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test -n '' 2023-10-11T11:14:56.605 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ get_image_id cluster1 mirror image_5 2023-10-11T11:14:56.605 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:14:56.605 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local pool=mirror 2023-10-11T11:14:56.605 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local image=image_5 2023-10-11T11:14:56.605 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2023-10-11T11:14:56.606 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-10-11T11:14:56.644 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image_id=13162090c09 2023-10-11T11:14:56.644 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-10-11T11:14:56.645 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.1 2023-10-11T11:14:56.746 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test_image_present cluster1 mirror image_5 present 13162090c09 2023-10-11T11:14:56.746 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:14:56.746 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:14:56.747 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_5 2023-10-11T11:14:56.747 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local test_state=present 2023-10-11T11:14:56.747 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image_id=13162090c09 2023-10-11T11:14:56.747 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_state=deleted 2023-10-11T11:14:56.747 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_image_id 2023-10-11T11:14:56.747 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ get_image_id cluster1 mirror image_5 2023-10-11T11:14:56.748 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:14:56.748 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local pool=mirror 2023-10-11T11:14:56.748 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local image=image_5 2023-10-11T11:14:56.748 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2023-10-11T11:14:56.748 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-10-11T11:14:56.789 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ current_image_id=13162090c09 2023-10-11T11:14:56.789 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test -n 13162090c09 2023-10-11T11:14:56.790 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test -z 13162090c09 -o 13162090c09 = 13162090c09 2023-10-11T11:14:56.790 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ current_state=present 2023-10-11T11:14:56.790 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test present = present 2023-10-11T11:14:56.790 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:14:56.790 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ disable_mirror cluster2 mirror image_5 2023-10-11T11:14:56.790 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:14:56.791 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:14:56.791 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_5 2023-10-11T11:14:56.791 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_5 2023-10-11T11:14:59.702 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi111.stderr:2023-10-11T11:14:59.701+0000 7f1e687ca700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x556984c77d40 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-10-11T11:14:59.702 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi111.stderr:2023-10-11T11:14:59.701+0000 7f1e6f7d8700 -1 rbd::mirror::ImageReplayer: 0x556984a07180 [3/ed1be0f3-19ac-46e3-a670-bd5d93bbe097] operator(): start failed: (85) Interrupted system call should be restarted 2023-10-11T11:15:26.649 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi111.stderr:2023-10-11T11:15:26.648+0000 7f2c29ed3700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55c1e975fc00 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=376 2023-10-11T11:15:26.650 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi111.stderr:2023-10-11T11:15:26.648+0000 7f2c29ed3700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55c1e975fc00 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=380, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=9a182971-2e50-451f-a839-7c08d3b6157c, primary_snap_id=178, last_copied_object_number=128, snap_seqs={376=18446744073709551614}] 2023-10-11T11:15:26.654 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:Mirroring disabled 2023-10-11T11:15:26.672 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_present cluster1 mirror image_5 deleted 2023-10-11T11:15:26.673 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:15:26.673 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:15:26.673 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_5 2023-10-11T11:15:26.673 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local state=deleted 2023-10-11T11:15:26.673 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image_id= 2023-10-11T11:15:26.673 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T11:15:26.674 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test -n '' 2023-10-11T11:15:26.674 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ get_image_id cluster1 mirror image_5 2023-10-11T11:15:26.674 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:15:26.674 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local pool=mirror 2023-10-11T11:15:26.674 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local image=image_5 2023-10-11T11:15:26.675 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2023-10-11T11:15:26.675 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-10-11T11:15:26.709 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd: error opening image image_5: (2) No such file or directory 2023-10-11T11:15:26.712 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image_id= 2023-10-11T11:15:26.712 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-10-11T11:15:26.713 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.1 2023-10-11T11:15:26.814 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test_image_present cluster1 mirror image_5 deleted '' 2023-10-11T11:15:26.814 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:15:26.815 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:15:26.815 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_5 2023-10-11T11:15:26.815 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local test_state=deleted 2023-10-11T11:15:26.815 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image_id= 2023-10-11T11:15:26.815 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_state=deleted 2023-10-11T11:15:26.815 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_image_id 2023-10-11T11:15:26.816 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ get_image_id cluster1 mirror image_5 2023-10-11T11:15:26.816 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:15:26.816 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local pool=mirror 2023-10-11T11:15:26.816 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local image=image_5 2023-10-11T11:15:26.816 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2023-10-11T11:15:26.817 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-10-11T11:15:26.853 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd: error opening image image_5: (2) No such file or directory 2023-10-11T11:15:26.856 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ current_image_id= 2023-10-11T11:15:26.857 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test -n '' 2023-10-11T11:15:26.857 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test deleted = deleted 2023-10-11T11:15:26.857 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:15:26.857 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ purge_snapshots cluster2 mirror image_5 2023-10-11T11:15:26.857 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:15:26.858 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:15:26.858 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_5 2023-10-11T11:15:26.858 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 snap purge mirror/image_5 2023-10-11T11:15:26.900 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_image_retry cluster2 mirror image_5 2023-10-11T11:15:26.900 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:15:26.900 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:15:26.900 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_5 2023-10-11T11:15:26.900 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0 1 2 4 8 16 32 2023-10-11T11:15:26.900 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0 2023-10-11T11:15:26.901 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_image cluster2 mirror image_5 2023-10-11T11:15:26.901 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:15:26.902 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:15:26.902 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_5 2023-10-11T11:15:26.902 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_5 2023-10-11T11:15:26.944 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 rm mirror/image_5 2023-10-11T11:15:27.229 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2023-10-11T11:15:27.232 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:15:27.233 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 10` 2023-10-11T11:15:27.233 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_6 2023-10-11T11:15:27.233 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ create_image cluster2 mirror image_6 512M 2023-10-11T11:15:27.233 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:15:27.233 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T11:15:27.233 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:15:27.234 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T11:15:27.234 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_6 2023-10-11T11:15:27.234 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T11:15:27.234 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local size=128 2023-10-11T11:15:27.234 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n 512M ']' 2023-10-11T11:15:27.235 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ size=512M 2023-10-11T11:15:27.235 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T11:15:27.235 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_6 2023-10-11T11:15:27.280 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ enable_mirror cluster2 mirror image_6 2023-10-11T11:15:27.281 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:15:27.281 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:15:27.281 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_6 2023-10-11T11:15:27.281 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local mode=snapshot 2023-10-11T11:15:27.281 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_6 snapshot 2023-10-11T11:15:27.661 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:Mirroring enabled 2023-10-11T11:15:27.671 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 info mirror/image_6 2023-10-11T11:15:27.708 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:rbd image 'image_6': 2023-10-11T11:15:27.708 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: size 512 MiB in 128 objects 2023-10-11T11:15:27.708 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: order 22 (4 MiB objects) 2023-10-11T11:15:27.708 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: snapshot_count: 1 2023-10-11T11:15:27.708 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: id: 1932e80adf8d 2023-10-11T11:15:27.709 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: block_name_prefix: rbd_data.1932e80adf8d 2023-10-11T11:15:27.709 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: format: 2 2023-10-11T11:15:27.709 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-10-11T11:15:27.709 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: op_features: 2023-10-11T11:15:27.710 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: flags: 2023-10-11T11:15:27.710 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: create_timestamp: Wed Oct 11 11:15:27 2023 2023-10-11T11:15:27.710 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: access_timestamp: Wed Oct 11 11:15:27 2023 2023-10-11T11:15:27.710 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: modify_timestamp: Wed Oct 11 11:15:27 2023 2023-10-11T11:15:27.710 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring state: enabled 2023-10-11T11:15:27.711 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring mode: snapshot 2023-10-11T11:15:27.711 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring global id: 1dd577b0-f202-4923-b5f9-8e4554fef3ab 2023-10-11T11:15:27.711 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring primary: true 2023-10-11T11:15:27.712 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ stress_write_image cluster2 mirror image_6 2023-10-11T11:15:27.712 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:15:27.712 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:15:27.713 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_6 2023-10-11T11:15:27.713 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-10-11T11:15:27.713 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local duration=12 2023-10-11T11:15:27.714 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ set +e 2023-10-11T11:15:27.714 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_6 --debug-rbd=20 --debug-journaler=20 2023-10-11T11:15:28.028 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:100 IOs, 59905024 bytes 2023-10-11T11:15:28.301 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:200 IOs, 119106560 bytes 2023-10-11T11:15:28.651 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:300 IOs, 227679232 bytes 2023-10-11T11:15:28.893 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:400 IOs, 301556736 bytes 2023-10-11T11:15:29.061 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:500 IOs, 340676608 bytes 2023-10-11T11:15:29.229 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:600 IOs, 377719808 bytes 2023-10-11T11:15:29.374 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:700 IOs, 401383424 bytes 2023-10-11T11:15:29.564 INFO:tasks.ceph.cluster2.osd.0.smithi111.stderr:2023-10-11T11:15:29.563+0000 7f80c3037700 -1 bdev(0x5555b6bee000 /var/lib/ceph/osd/cluster2-0/block) _aio_thread got r=-28 ((28) No space left on device) 2023-10-11T11:15:29.566 INFO:tasks.ceph.cluster2.osd.0.smithi111.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-6661-g4bf94f59/rpm/el8/BUILD/ceph-18.0.0-6661-g4bf94f59/src/blk/kernel/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7f80c3037700 time 2023-10-11T11:15:29.564006+0000 2023-10-11T11:15:29.566 INFO:tasks.ceph.cluster2.osd.0.smithi111.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-6661-g4bf94f59/rpm/el8/BUILD/ceph-18.0.0-6661-g4bf94f59/src/blk/kernel/KernelDevice.cc: 633: ceph_abort_msg("Unexpected IO error. This may suggest a hardware issue. Please check your kernel log!") 2023-10-11T11:15:29.567 INFO:tasks.ceph.cluster2.osd.0.smithi111.stderr: ceph version 18.0.0-6661-g4bf94f59 (4bf94f59187ed7eba271559c0898cec750ad93d5) reef (dev) 2023-10-11T11:15:29.567 INFO:tasks.ceph.cluster2.osd.0.smithi111.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xd7) [0x5555b2d8a7bf] 2023-10-11T11:15:29.567 INFO:tasks.ceph.cluster2.osd.0.smithi111.stderr: 2: (KernelDevice::_aio_thread()+0x1686) [0x5555b39ee836] 2023-10-11T11:15:29.568 INFO:tasks.ceph.cluster2.osd.0.smithi111.stderr: 3: (KernelDevice::AioCompletionThread::entry()+0x11) [0x5555b39f4251] 2023-10-11T11:15:29.568 INFO:tasks.ceph.cluster2.osd.0.smithi111.stderr: 4: /lib64/libpthread.so.0(+0x81cf) [0x7f80d5b5e1cf] 2023-10-11T11:15:29.568 INFO:tasks.ceph.cluster2.osd.0.smithi111.stderr: 5: clone() 2023-10-11T11:15:29.568 INFO:tasks.ceph.cluster2.osd.0.smithi111.stderr:2023-10-11T11:15:29.565+0000 7f80c3037700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-6661-g4bf94f59/rpm/el8/BUILD/ceph-18.0.0-6661-g4bf94f59/src/blk/kernel/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7f80c3037700 time 2023-10-11T11:15:29.564006+0000 2023-10-11T11:15:29.569 INFO:tasks.ceph.cluster2.osd.0.smithi111.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-6661-g4bf94f59/rpm/el8/BUILD/ceph-18.0.0-6661-g4bf94f59/src/blk/kernel/KernelDevice.cc: 633: ceph_abort_msg("Unexpected IO error. This may suggest a hardware issue. Please check your kernel log!") 2023-10-11T11:15:29.569 INFO:tasks.ceph.cluster2.osd.0.smithi111.stderr: 2023-10-11T11:15:29.569 INFO:tasks.ceph.cluster2.osd.0.smithi111.stderr: ceph version 18.0.0-6661-g4bf94f59 (4bf94f59187ed7eba271559c0898cec750ad93d5) reef (dev) 2023-10-11T11:15:29.569 INFO:tasks.ceph.cluster2.osd.0.smithi111.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xd7) [0x5555b2d8a7bf] 2023-10-11T11:15:29.570 INFO:tasks.ceph.cluster2.osd.0.smithi111.stderr: 2: (KernelDevice::_aio_thread()+0x1686) [0x5555b39ee836] 2023-10-11T11:15:29.570 INFO:tasks.ceph.cluster2.osd.0.smithi111.stderr: 3: (KernelDevice::AioCompletionThread::entry()+0x11) [0x5555b39f4251] 2023-10-11T11:15:29.570 INFO:tasks.ceph.cluster2.osd.0.smithi111.stderr: 4: /lib64/libpthread.so.0(+0x81cf) [0x7f80d5b5e1cf] 2023-10-11T11:15:29.571 INFO:tasks.ceph.cluster2.osd.0.smithi111.stderr: 5: clone() 2023-10-11T11:15:29.571 INFO:tasks.ceph.cluster2.osd.0.smithi111.stderr: 2023-10-11T11:15:29.571 INFO:tasks.ceph.cluster2.osd.0.smithi111.stderr:*** Caught signal (Aborted) ** 2023-10-11T11:15:29.571 INFO:tasks.ceph.cluster2.osd.0.smithi111.stderr: in thread 7f80c3037700 thread_name:bstore_aio 2023-10-11T11:15:29.572 INFO:tasks.ceph.cluster2.osd.0.smithi111.stderr: ceph version 18.0.0-6661-g4bf94f59 (4bf94f59187ed7eba271559c0898cec750ad93d5) reef (dev) 2023-10-11T11:15:29.572 INFO:tasks.ceph.cluster2.osd.0.smithi111.stderr: 1: /lib64/libpthread.so.0(+0x12ce0) [0x7f80d5b68ce0] 2023-10-11T11:15:29.573 INFO:tasks.ceph.cluster2.osd.0.smithi111.stderr: 2: gsignal() 2023-10-11T11:15:29.573 INFO:tasks.ceph.cluster2.osd.0.smithi111.stderr: 3: abort() 2023-10-11T11:15:29.573 INFO:tasks.ceph.cluster2.osd.0.smithi111.stderr: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x197) [0x5555b2d8a87f] 2023-10-11T11:15:29.574 INFO:tasks.ceph.cluster2.osd.0.smithi111.stderr: 5: (KernelDevice::_aio_thread()+0x1686) [0x5555b39ee836] 2023-10-11T11:15:29.574 INFO:tasks.ceph.cluster2.osd.0.smithi111.stderr: 6: (KernelDevice::AioCompletionThread::entry()+0x11) [0x5555b39f4251] 2023-10-11T11:15:29.574 INFO:tasks.ceph.cluster2.osd.0.smithi111.stderr: 7: /lib64/libpthread.so.0(+0x81cf) [0x7f80d5b5e1cf] 2023-10-11T11:15:29.574 INFO:tasks.ceph.cluster2.osd.0.smithi111.stderr: 8: clone() 2023-10-11T11:15:29.575 INFO:tasks.ceph.cluster2.osd.0.smithi111.stderr:2023-10-11T11:15:29.567+0000 7f80c3037700 -1 *** Caught signal (Aborted) ** 2023-10-11T11:15:29.575 INFO:tasks.ceph.cluster2.osd.0.smithi111.stderr: in thread 7f80c3037700 thread_name:bstore_aio 2023-10-11T11:15:29.575 INFO:tasks.ceph.cluster2.osd.0.smithi111.stderr: 2023-10-11T11:15:29.575 INFO:tasks.ceph.cluster2.osd.0.smithi111.stderr: ceph version 18.0.0-6661-g4bf94f59 (4bf94f59187ed7eba271559c0898cec750ad93d5) reef (dev) 2023-10-11T11:15:29.576 INFO:tasks.ceph.cluster2.osd.0.smithi111.stderr: 1: /lib64/libpthread.so.0(+0x12ce0) [0x7f80d5b68ce0] 2023-10-11T11:15:29.576 INFO:tasks.ceph.cluster2.osd.0.smithi111.stderr: 2: gsignal() 2023-10-11T11:15:29.576 INFO:tasks.ceph.cluster2.osd.0.smithi111.stderr: 3: abort() 2023-10-11T11:15:29.577 INFO:tasks.ceph.cluster2.osd.0.smithi111.stderr: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x197) [0x5555b2d8a87f] 2023-10-11T11:15:29.577 INFO:tasks.ceph.cluster2.osd.0.smithi111.stderr: 5: (KernelDevice::_aio_thread()+0x1686) [0x5555b39ee836] 2023-10-11T11:15:29.577 INFO:tasks.ceph.cluster2.osd.0.smithi111.stderr: 6: (KernelDevice::AioCompletionThread::entry()+0x11) [0x5555b39f4251] 2023-10-11T11:15:29.577 INFO:tasks.ceph.cluster2.osd.0.smithi111.stderr: 7: /lib64/libpthread.so.0(+0x81cf) [0x7f80d5b5e1cf] 2023-10-11T11:15:29.578 INFO:tasks.ceph.cluster2.osd.0.smithi111.stderr: 8: clone() 2023-10-11T11:15:29.578 INFO:tasks.ceph.cluster2.osd.0.smithi111.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-10-11T11:15:29.578 INFO:tasks.ceph.cluster2.osd.0.smithi111.stderr: 2023-10-11T11:15:29.579 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:800 IOs, 455989248 bytes 2023-10-11T11:15:29.584 INFO:tasks.ceph.cluster2.osd.0.smithi111.stderr: -1165> 2023-10-11T11:15:29.563+0000 7f80c3037700 -1 bdev(0x5555b6bee000 /var/lib/ceph/osd/cluster2-0/block) _aio_thread got r=-28 ((28) No space left on device) 2023-10-11T11:15:29.585 INFO:tasks.ceph.cluster2.osd.0.smithi111.stderr: -437> 2023-10-11T11:15:29.565+0000 7f80c3037700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-6661-g4bf94f59/rpm/el8/BUILD/ceph-18.0.0-6661-g4bf94f59/src/blk/kernel/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7f80c3037700 time 2023-10-11T11:15:29.564006+0000 2023-10-11T11:15:29.585 INFO:tasks.ceph.cluster2.osd.0.smithi111.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-6661-g4bf94f59/rpm/el8/BUILD/ceph-18.0.0-6661-g4bf94f59/src/blk/kernel/KernelDevice.cc: 633: ceph_abort_msg("Unexpected IO error. This may suggest a hardware issue. Please check your kernel log!") 2023-10-11T11:15:29.586 INFO:tasks.ceph.cluster2.osd.0.smithi111.stderr: 2023-10-11T11:15:29.586 INFO:tasks.ceph.cluster2.osd.0.smithi111.stderr: ceph version 18.0.0-6661-g4bf94f59 (4bf94f59187ed7eba271559c0898cec750ad93d5) reef (dev) 2023-10-11T11:15:29.586 INFO:tasks.ceph.cluster2.osd.0.smithi111.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xd7) [0x5555b2d8a7bf] 2023-10-11T11:15:29.586 INFO:tasks.ceph.cluster2.osd.0.smithi111.stderr: 2: (KernelDevice::_aio_thread()+0x1686) [0x5555b39ee836] 2023-10-11T11:15:29.587 INFO:tasks.ceph.cluster2.osd.0.smithi111.stderr: 3: (KernelDevice::AioCompletionThread::entry()+0x11) [0x5555b39f4251] 2023-10-11T11:15:29.587 INFO:tasks.ceph.cluster2.osd.0.smithi111.stderr: 4: /lib64/libpthread.so.0(+0x81cf) [0x7f80d5b5e1cf] 2023-10-11T11:15:29.587 INFO:tasks.ceph.cluster2.osd.0.smithi111.stderr: 5: clone() 2023-10-11T11:15:29.587 INFO:tasks.ceph.cluster2.osd.0.smithi111.stderr: 2023-10-11T11:15:29.588 INFO:tasks.ceph.cluster2.osd.0.smithi111.stderr: -7> 2023-10-11T11:15:29.567+0000 7f80c3037700 -1 *** Caught signal (Aborted) ** 2023-10-11T11:15:29.588 INFO:tasks.ceph.cluster2.osd.0.smithi111.stderr: in thread 7f80c3037700 thread_name:bstore_aio 2023-10-11T11:15:29.588 INFO:tasks.ceph.cluster2.osd.0.smithi111.stderr: 2023-10-11T11:15:29.589 INFO:tasks.ceph.cluster2.osd.0.smithi111.stderr: ceph version 18.0.0-6661-g4bf94f59 (4bf94f59187ed7eba271559c0898cec750ad93d5) reef (dev) 2023-10-11T11:15:29.589 INFO:tasks.ceph.cluster2.osd.0.smithi111.stderr: 1: /lib64/libpthread.so.0(+0x12ce0) [0x7f80d5b68ce0] 2023-10-11T11:15:29.589 INFO:tasks.ceph.cluster2.osd.0.smithi111.stderr: 2: gsignal() 2023-10-11T11:15:29.589 INFO:tasks.ceph.cluster2.osd.0.smithi111.stderr: 3: abort() 2023-10-11T11:15:29.590 INFO:tasks.ceph.cluster2.osd.0.smithi111.stderr: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x197) [0x5555b2d8a87f] 2023-10-11T11:15:29.590 INFO:tasks.ceph.cluster2.osd.0.smithi111.stderr: 5: (KernelDevice::_aio_thread()+0x1686) [0x5555b39ee836] 2023-10-11T11:15:29.590 INFO:tasks.ceph.cluster2.osd.0.smithi111.stderr: 6: (KernelDevice::AioCompletionThread::entry()+0x11) [0x5555b39f4251] 2023-10-11T11:15:29.590 INFO:tasks.ceph.cluster2.osd.0.smithi111.stderr: 7: /lib64/libpthread.so.0(+0x81cf) [0x7f80d5b5e1cf] 2023-10-11T11:15:29.591 INFO:tasks.ceph.cluster2.osd.0.smithi111.stderr: 8: clone() 2023-10-11T11:15:29.591 INFO:tasks.ceph.cluster2.osd.0.smithi111.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-10-11T11:15:29.591 INFO:tasks.ceph.cluster2.osd.0.smithi111.stderr: 2023-10-11T11:15:29.594 INFO:tasks.ceph.cluster2.osd.0.smithi111.stderr: -2166> 2023-10-11T11:15:29.563+0000 7f80c3037700 -1 bdev(0x5555b6bee000 /var/lib/ceph/osd/cluster2-0/block) _aio_thread got r=-28 ((28) No space left on device) 2023-10-11T11:15:29.595 INFO:tasks.ceph.cluster2.osd.0.smithi111.stderr: -1438> 2023-10-11T11:15:29.565+0000 7f80c3037700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-6661-g4bf94f59/rpm/el8/BUILD/ceph-18.0.0-6661-g4bf94f59/src/blk/kernel/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7f80c3037700 time 2023-10-11T11:15:29.564006+0000 2023-10-11T11:15:29.596 INFO:tasks.ceph.cluster2.osd.0.smithi111.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-6661-g4bf94f59/rpm/el8/BUILD/ceph-18.0.0-6661-g4bf94f59/src/blk/kernel/KernelDevice.cc: 633: ceph_abort_msg("Unexpected IO error. This may suggest a hardware issue. Please check your kernel log!") 2023-10-11T11:15:29.596 INFO:tasks.ceph.cluster2.osd.0.smithi111.stderr: 2023-10-11T11:15:29.596 INFO:tasks.ceph.cluster2.osd.0.smithi111.stderr: ceph version 18.0.0-6661-g4bf94f59 (4bf94f59187ed7eba271559c0898cec750ad93d5) reef (dev) 2023-10-11T11:15:29.596 INFO:tasks.ceph.cluster2.osd.0.smithi111.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xd7) [0x5555b2d8a7bf] 2023-10-11T11:15:29.597 INFO:tasks.ceph.cluster2.osd.0.smithi111.stderr: 2: (KernelDevice::_aio_thread()+0x1686) [0x5555b39ee836] 2023-10-11T11:15:29.597 INFO:tasks.ceph.cluster2.osd.0.smithi111.stderr: 3: (KernelDevice::AioCompletionThread::entry()+0x11) [0x5555b39f4251] 2023-10-11T11:15:29.597 INFO:tasks.ceph.cluster2.osd.0.smithi111.stderr: 4: /lib64/libpthread.so.0(+0x81cf) [0x7f80d5b5e1cf] 2023-10-11T11:15:29.597 INFO:tasks.ceph.cluster2.osd.0.smithi111.stderr: 5: clone() 2023-10-11T11:15:29.598 INFO:tasks.ceph.cluster2.osd.0.smithi111.stderr: 2023-10-11T11:15:29.598 INFO:tasks.ceph.cluster2.osd.0.smithi111.stderr: -1008> 2023-10-11T11:15:29.567+0000 7f80c3037700 -1 *** Caught signal (Aborted) ** 2023-10-11T11:15:29.598 INFO:tasks.ceph.cluster2.osd.0.smithi111.stderr: in thread 7f80c3037700 thread_name:bstore_aio 2023-10-11T11:15:29.598 INFO:tasks.ceph.cluster2.osd.0.smithi111.stderr: 2023-10-11T11:15:29.599 INFO:tasks.ceph.cluster2.osd.0.smithi111.stderr: ceph version 18.0.0-6661-g4bf94f59 (4bf94f59187ed7eba271559c0898cec750ad93d5) reef (dev) 2023-10-11T11:15:29.599 INFO:tasks.ceph.cluster2.osd.0.smithi111.stderr: 1: /lib64/libpthread.so.0(+0x12ce0) [0x7f80d5b68ce0] 2023-10-11T11:15:29.599 INFO:tasks.ceph.cluster2.osd.0.smithi111.stderr: 2: gsignal() 2023-10-11T11:15:29.599 INFO:tasks.ceph.cluster2.osd.0.smithi111.stderr: 3: abort() 2023-10-11T11:15:29.600 INFO:tasks.ceph.cluster2.osd.0.smithi111.stderr: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x197) [0x5555b2d8a87f] 2023-10-11T11:15:29.600 INFO:tasks.ceph.cluster2.osd.0.smithi111.stderr: 5: (KernelDevice::_aio_thread()+0x1686) [0x5555b39ee836] 2023-10-11T11:15:29.600 INFO:tasks.ceph.cluster2.osd.0.smithi111.stderr: 6: (KernelDevice::AioCompletionThread::entry()+0x11) [0x5555b39f4251] 2023-10-11T11:15:29.600 INFO:tasks.ceph.cluster2.osd.0.smithi111.stderr: 7: /lib64/libpthread.so.0(+0x81cf) [0x7f80d5b5e1cf] 2023-10-11T11:15:29.601 INFO:tasks.ceph.cluster2.osd.0.smithi111.stderr: 8: clone() 2023-10-11T11:15:29.601 INFO:tasks.ceph.cluster2.osd.0.smithi111.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-10-11T11:15:29.601 INFO:tasks.ceph.cluster2.osd.0.smithi111.stderr: 2023-10-11T11:15:29.867 INFO:tasks.ceph.cluster2.osd.0.smithi111.stderr:daemon-helper: command failed with exit status 5 2023-10-11T11:15:31.811 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:900 IOs, 491809792 bytes 2023-10-11T11:15:31.917 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1000 IOs, 514625536 bytes 2023-10-11T11:15:32.102 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1100 IOs, 553603072 bytes 2023-10-11T11:15:32.169 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~0s 2023-10-11T11:15:32.326 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1200 IOs, 599101440 bytes 2023-10-11T11:15:33.727 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1300 IOs, 657874944 bytes 2023-10-11T11:15:33.894 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1400 IOs, 735451136 bytes 2023-10-11T11:15:34.028 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1500 IOs, 784048128 bytes 2023-10-11T11:15:34.113 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1600 IOs, 784677888 bytes 2023-10-11T11:15:34.265 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1700 IOs, 835110912 bytes 2023-10-11T11:15:34.403 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1800 IOs, 887028736 bytes 2023-10-11T11:15:34.502 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1900 IOs, 904190976 bytes 2023-10-11T11:15:34.618 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2000 IOs, 927430656 bytes 2023-10-11T11:15:34.690 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2100 IOs, 928290816 bytes 2023-10-11T11:15:34.779 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2200 IOs, 934777856 bytes 2023-10-11T11:15:34.941 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2300 IOs, 1000223744 bytes 2023-10-11T11:15:35.036 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2400 IOs, 1022770176 bytes 2023-10-11T11:15:35.196 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2500 IOs, 1067667456 bytes 2023-10-11T11:15:35.283 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2600 IOs, 1073657856 bytes 2023-10-11T11:15:35.454 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2700 IOs, 1150404608 bytes 2023-10-11T11:15:35.579 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2800 IOs, 1182360576 bytes 2023-10-11T11:15:35.721 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2900 IOs, 1228272640 bytes 2023-10-11T11:15:35.829 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3000 IOs, 1254037504 bytes 2023-10-11T11:15:35.925 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3100 IOs, 1254663168 bytes 2023-10-11T11:15:36.072 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3200 IOs, 1312791552 bytes 2023-10-11T11:15:36.183 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3300 IOs, 1336665088 bytes 2023-10-11T11:15:36.370 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3400 IOs, 1418140672 bytes 2023-10-11T11:15:36.512 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3500 IOs, 1462021120 bytes 2023-10-11T11:15:36.597 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3600 IOs, 1471955968 bytes 2023-10-11T11:15:36.684 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3700 IOs, 1489213440 bytes 2023-10-11T11:15:36.808 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3800 IOs, 1533657088 bytes 2023-10-11T11:15:36.899 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3900 IOs, 1545640960 bytes 2023-10-11T11:15:37.108 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4000 IOs, 1666556928 bytes 2023-10-11T11:15:37.218 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4100 IOs, 1677976576 bytes 2023-10-11T11:15:37.383 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4200 IOs, 1754820608 bytes 2023-10-11T11:15:37.494 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4300 IOs, 1780002816 bytes 2023-10-11T11:15:37.574 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~5s 2023-10-11T11:15:37.725 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4400 IOs, 1888120832 bytes 2023-10-11T11:15:37.873 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4500 IOs, 1944562688 bytes 2023-10-11T11:15:37.973 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4600 IOs, 1962279936 bytes 2023-10-11T11:15:38.115 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4700 IOs, 2014410752 bytes 2023-10-11T11:15:38.241 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4800 IOs, 2034909184 bytes 2023-10-11T11:15:38.363 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4900 IOs, 2070905856 bytes 2023-10-11T11:15:38.455 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5000 IOs, 2082461696 bytes 2023-10-11T11:15:38.642 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5100 IOs, 2146507776 bytes 2023-10-11T11:15:38.807 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5200 IOs, 2224970752 bytes 2023-10-11T11:15:39.027 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5300 IOs, 2311268352 bytes 2023-10-11T11:15:39.176 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5400 IOs, 2374579200 bytes 2023-10-11T11:15:39.272 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5500 IOs, 2387456000 bytes 2023-10-11T11:15:39.352 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5600 IOs, 2392894464 bytes 2023-10-11T11:15:39.482 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5700 IOs, 2426941440 bytes 2023-10-11T11:15:39.606 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5800 IOs, 2466719744 bytes 2023-10-11T11:15:39.718 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ error_code=124 2023-10-11T11:15:39.718 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ set -e 2023-10-11T11:15:39.718 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' 124 -eq 124 ']' 2023-10-11T11:15:39.718 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:15:39.719 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_present cluster1 mirror image_6 present 2023-10-11T11:15:39.719 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:15:39.719 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:15:39.719 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_6 2023-10-11T11:15:39.719 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local state=present 2023-10-11T11:15:39.720 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image_id= 2023-10-11T11:15:39.720 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T11:15:39.720 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test -n '' 2023-10-11T11:15:39.720 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ get_image_id cluster1 mirror image_6 2023-10-11T11:15:39.720 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:15:39.721 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local pool=mirror 2023-10-11T11:15:39.721 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local image=image_6 2023-10-11T11:15:39.721 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2023-10-11T11:15:39.721 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-10-11T11:15:39.760 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image_id=1316db2d9378 2023-10-11T11:15:39.761 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-10-11T11:15:39.761 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.1 2023-10-11T11:15:39.862 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test_image_present cluster1 mirror image_6 present 1316db2d9378 2023-10-11T11:15:39.863 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:15:39.863 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:15:39.863 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_6 2023-10-11T11:15:39.863 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local test_state=present 2023-10-11T11:15:39.863 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image_id=1316db2d9378 2023-10-11T11:15:39.864 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_state=deleted 2023-10-11T11:15:39.864 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_image_id 2023-10-11T11:15:39.864 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ get_image_id cluster1 mirror image_6 2023-10-11T11:15:39.864 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:15:39.864 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local pool=mirror 2023-10-11T11:15:39.865 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local image=image_6 2023-10-11T11:15:39.865 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2023-10-11T11:15:39.865 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-10-11T11:15:39.905 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ current_image_id=1316db2d9378 2023-10-11T11:15:39.905 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test -n 1316db2d9378 2023-10-11T11:15:39.905 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test -z 1316db2d9378 -o 1316db2d9378 = 1316db2d9378 2023-10-11T11:15:39.906 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ current_state=present 2023-10-11T11:15:39.906 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test present = present 2023-10-11T11:15:39.906 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:15:39.906 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ disable_mirror cluster2 mirror image_6 2023-10-11T11:15:39.906 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:15:39.906 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:15:39.907 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_6 2023-10-11T11:15:39.907 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_6 2023-10-11T11:15:42.977 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~11s 2023-10-11T11:15:47.604 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.4731196086890185) > weight (0.25) 2023-10-11T11:15:47.604 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.8992593533943848) > weight (0.25) 2023-10-11T11:15:47.604 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.7113757430530814) > weight (0.25) 2023-10-11T11:15:47.605 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.37816463919947596) > weight (0.25) 2023-10-11T11:15:47.605 INFO:tasks.rbd_mirror_thrash:waiting for 108 secs before thrashing 2023-10-11T11:15:48.380 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~16s 2023-10-11T11:15:53.783 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~22s 2023-10-11T11:15:59.186 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~27s 2023-10-11T11:15:59.702 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi111.stderr:2023-10-11T11:15:59.700+0000 7f1e687ca700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x556983bc5380 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-10-11T11:15:59.702 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi111.stderr:2023-10-11T11:15:59.700+0000 7f1e6efd7700 -1 rbd::mirror::ImageReplayer: 0x5569846e5900 [3/1dd577b0-f202-4923-b5f9-8e4554fef3ab] operator(): start failed: (85) Interrupted system call should be restarted 2023-10-11T11:16:04.590 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~32s 2023-10-11T11:16:09.994 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~38s 2023-10-11T11:16:10.658 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi111.stderr:2023-10-11T11:16:10.657+0000 7f2c29ed3700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55c1e89f0800 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=378 2023-10-11T11:16:10.658 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi111.stderr:2023-10-11T11:16:10.657+0000 7f2c29ed3700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55c1e89f0800 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=382, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=9a182971-2e50-451f-a839-7c08d3b6157c, primary_snap_id=17a, last_copied_object_number=128, snap_seqs={378=18446744073709551614}] 2023-10-11T11:16:10.668 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:Mirroring disabled 2023-10-11T11:16:10.680 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_present cluster1 mirror image_6 deleted 2023-10-11T11:16:10.680 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:16:10.680 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:16:10.680 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_6 2023-10-11T11:16:10.680 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local state=deleted 2023-10-11T11:16:10.681 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image_id= 2023-10-11T11:16:10.681 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T11:16:10.681 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test -n '' 2023-10-11T11:16:10.681 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ get_image_id cluster1 mirror image_6 2023-10-11T11:16:10.681 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:16:10.681 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local pool=mirror 2023-10-11T11:16:10.682 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local image=image_6 2023-10-11T11:16:10.682 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2023-10-11T11:16:10.682 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-10-11T11:16:10.715 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd: error opening image image_6: (2) No such file or directory 2023-10-11T11:16:10.719 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image_id= 2023-10-11T11:16:10.719 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-10-11T11:16:10.719 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.1 2023-10-11T11:16:10.821 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test_image_present cluster1 mirror image_6 deleted '' 2023-10-11T11:16:10.821 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:16:10.821 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:16:10.821 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_6 2023-10-11T11:16:10.821 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local test_state=deleted 2023-10-11T11:16:10.822 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image_id= 2023-10-11T11:16:10.822 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_state=deleted 2023-10-11T11:16:10.822 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_image_id 2023-10-11T11:16:10.822 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ get_image_id cluster1 mirror image_6 2023-10-11T11:16:10.822 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:16:10.823 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local pool=mirror 2023-10-11T11:16:10.823 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local image=image_6 2023-10-11T11:16:10.823 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2023-10-11T11:16:10.823 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-10-11T11:16:10.856 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd: error opening image image_6: (2) No such file or directory 2023-10-11T11:16:10.859 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ current_image_id= 2023-10-11T11:16:10.859 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test -n '' 2023-10-11T11:16:10.860 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test deleted = deleted 2023-10-11T11:16:10.860 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:16:10.860 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ purge_snapshots cluster2 mirror image_6 2023-10-11T11:16:10.860 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:16:10.860 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:16:10.861 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_6 2023-10-11T11:16:10.861 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 snap purge mirror/image_6 2023-10-11T11:16:10.904 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_image_retry cluster2 mirror image_6 2023-10-11T11:16:10.904 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:16:10.904 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:16:10.904 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_6 2023-10-11T11:16:10.904 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0 1 2 4 8 16 32 2023-10-11T11:16:10.905 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0 2023-10-11T11:16:10.905 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_image cluster2 mirror image_6 2023-10-11T11:16:10.905 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:16:10.906 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:16:10.906 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_6 2023-10-11T11:16:10.906 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_6 2023-10-11T11:16:10.947 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 rm mirror/image_6 2023-10-11T11:16:11.173 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2023-10-11T11:16:11.177 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:16:11.178 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 10` 2023-10-11T11:16:11.178 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_7 2023-10-11T11:16:11.178 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ create_image cluster2 mirror image_7 512M 2023-10-11T11:16:11.178 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:16:11.178 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T11:16:11.179 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:16:11.179 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T11:16:11.179 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_7 2023-10-11T11:16:11.179 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T11:16:11.179 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local size=128 2023-10-11T11:16:11.179 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n 512M ']' 2023-10-11T11:16:11.180 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ size=512M 2023-10-11T11:16:11.180 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T11:16:11.180 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_7 2023-10-11T11:16:11.226 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ enable_mirror cluster2 mirror image_7 2023-10-11T11:16:11.226 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:16:11.226 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:16:11.226 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_7 2023-10-11T11:16:11.227 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local mode=snapshot 2023-10-11T11:16:11.227 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_7 snapshot 2023-10-11T11:16:11.681 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:Mirroring enabled 2023-10-11T11:16:11.691 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 info mirror/image_7 2023-10-11T11:16:11.726 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:rbd image 'image_7': 2023-10-11T11:16:11.726 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: size 512 MiB in 128 objects 2023-10-11T11:16:11.727 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: order 22 (4 MiB objects) 2023-10-11T11:16:11.727 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: snapshot_count: 1 2023-10-11T11:16:11.727 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: id: 194a24a19d8f 2023-10-11T11:16:11.727 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: block_name_prefix: rbd_data.194a24a19d8f 2023-10-11T11:16:11.727 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: format: 2 2023-10-11T11:16:11.728 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-10-11T11:16:11.728 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: op_features: 2023-10-11T11:16:11.728 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: flags: 2023-10-11T11:16:11.728 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: create_timestamp: Wed Oct 11 11:16:11 2023 2023-10-11T11:16:11.728 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: access_timestamp: Wed Oct 11 11:16:11 2023 2023-10-11T11:16:11.728 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: modify_timestamp: Wed Oct 11 11:16:11 2023 2023-10-11T11:16:11.729 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring state: enabled 2023-10-11T11:16:11.729 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring mode: snapshot 2023-10-11T11:16:11.729 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring global id: b48a49ee-b365-4773-a4fc-2556af272547 2023-10-11T11:16:11.729 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring primary: true 2023-10-11T11:16:11.730 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ stress_write_image cluster2 mirror image_7 2023-10-11T11:16:11.730 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:16:11.730 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:16:11.730 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_7 2023-10-11T11:16:11.731 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-10-11T11:16:11.731 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local duration=6 2023-10-11T11:16:11.731 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ set +e 2023-10-11T11:16:11.732 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_7 --debug-rbd=20 --debug-journaler=20 2023-10-11T11:16:11.951 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:100 IOs, 17293312 bytes 2023-10-11T11:16:12.145 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:200 IOs, 119079936 bytes 2023-10-11T11:16:12.281 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:300 IOs, 163914752 bytes 2023-10-11T11:16:12.406 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:400 IOs, 193633280 bytes 2023-10-11T11:16:12.557 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:500 IOs, 256020480 bytes 2023-10-11T11:16:12.631 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:600 IOs, 256649216 bytes 2023-10-11T11:16:12.763 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:700 IOs, 317451264 bytes 2023-10-11T11:16:12.877 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:800 IOs, 349879296 bytes 2023-10-11T11:16:13.005 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:900 IOs, 402595840 bytes 2023-10-11T11:16:13.131 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1000 IOs, 453650432 bytes 2023-10-11T11:16:13.245 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1100 IOs, 474343424 bytes 2023-10-11T11:16:13.355 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1200 IOs, 516470784 bytes 2023-10-11T11:16:13.507 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1300 IOs, 590882816 bytes 2023-10-11T11:16:13.672 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1400 IOs, 655191040 bytes 2023-10-11T11:16:13.744 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1500 IOs, 680304640 bytes 2023-10-11T11:16:13.916 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1600 IOs, 732019712 bytes 2023-10-11T11:16:14.051 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1700 IOs, 778739712 bytes 2023-10-11T11:16:14.130 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1800 IOs, 801325056 bytes 2023-10-11T11:16:14.224 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1900 IOs, 818162688 bytes 2023-10-11T11:16:14.314 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2000 IOs, 841493504 bytes 2023-10-11T11:16:14.479 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2100 IOs, 923735040 bytes 2023-10-11T11:16:14.591 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2200 IOs, 968194048 bytes 2023-10-11T11:16:14.693 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2300 IOs, 992334848 bytes 2023-10-11T11:16:14.768 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2400 IOs, 992950272 bytes 2023-10-11T11:16:15.004 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2500 IOs, 1087386624 bytes 2023-10-11T11:16:15.098 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2600 IOs, 1103157248 bytes 2023-10-11T11:16:15.191 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2700 IOs, 1125470208 bytes 2023-10-11T11:16:15.296 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2800 IOs, 1156992000 bytes 2023-10-11T11:16:15.397 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~43s 2023-10-11T11:16:15.441 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2900 IOs, 1214261248 bytes 2023-10-11T11:16:15.546 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3000 IOs, 1254572032 bytes 2023-10-11T11:16:15.649 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3100 IOs, 1270182912 bytes 2023-10-11T11:16:15.768 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3200 IOs, 1312973824 bytes 2023-10-11T11:16:15.874 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3300 IOs, 1353286656 bytes 2023-10-11T11:16:16.053 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3400 IOs, 1442862080 bytes 2023-10-11T11:16:16.165 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3500 IOs, 1474252800 bytes 2023-10-11T11:16:16.254 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3600 IOs, 1480298496 bytes 2023-10-11T11:16:16.411 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3700 IOs, 1533448192 bytes 2023-10-11T11:16:16.561 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3800 IOs, 1595012096 bytes 2023-10-11T11:16:16.723 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3900 IOs, 1663010816 bytes 2023-10-11T11:16:16.865 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4000 IOs, 1708275712 bytes 2023-10-11T11:16:16.993 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4100 IOs, 1763646464 bytes 2023-10-11T11:16:17.104 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4200 IOs, 1808943104 bytes 2023-10-11T11:16:17.214 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4300 IOs, 1835150336 bytes 2023-10-11T11:16:17.356 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4400 IOs, 1902582784 bytes 2023-10-11T11:16:17.470 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4500 IOs, 1940896768 bytes 2023-10-11T11:16:17.606 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4600 IOs, 1972868096 bytes 2023-10-11T11:16:17.729 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4700 IOs, 2045333504 bytes 2023-10-11T11:16:17.738 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ error_code=124 2023-10-11T11:16:17.738 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ set -e 2023-10-11T11:16:17.738 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' 124 -eq 124 ']' 2023-10-11T11:16:17.739 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:16:17.739 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_present cluster1 mirror image_7 present 2023-10-11T11:16:17.739 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:16:17.739 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:16:17.739 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_7 2023-10-11T11:16:17.739 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local state=present 2023-10-11T11:16:17.740 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image_id= 2023-10-11T11:16:17.740 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T11:16:17.740 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test -n '' 2023-10-11T11:16:17.740 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ get_image_id cluster1 mirror image_7 2023-10-11T11:16:17.740 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:16:17.741 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local pool=mirror 2023-10-11T11:16:17.741 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local image=image_7 2023-10-11T11:16:17.742 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2023-10-11T11:16:17.742 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-10-11T11:16:17.794 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image_id=13169865e756 2023-10-11T11:16:17.794 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-10-11T11:16:17.794 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.1 2023-10-11T11:16:17.895 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test_image_present cluster1 mirror image_7 present 13169865e756 2023-10-11T11:16:17.896 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:16:17.896 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:16:17.896 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_7 2023-10-11T11:16:17.896 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local test_state=present 2023-10-11T11:16:17.896 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image_id=13169865e756 2023-10-11T11:16:17.897 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_state=deleted 2023-10-11T11:16:17.897 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_image_id 2023-10-11T11:16:17.897 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ get_image_id cluster1 mirror image_7 2023-10-11T11:16:17.897 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:16:17.897 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local pool=mirror 2023-10-11T11:16:17.897 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local image=image_7 2023-10-11T11:16:17.898 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2023-10-11T11:16:17.898 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-10-11T11:16:17.941 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ current_image_id=13169865e756 2023-10-11T11:16:17.941 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test -n 13169865e756 2023-10-11T11:16:17.941 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test -z 13169865e756 -o 13169865e756 = 13169865e756 2023-10-11T11:16:17.941 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ current_state=present 2023-10-11T11:16:17.941 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test present = present 2023-10-11T11:16:17.942 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:16:17.942 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ disable_mirror cluster2 mirror image_7 2023-10-11T11:16:17.942 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:16:17.942 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:16:17.942 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_7 2023-10-11T11:16:17.943 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_7 2023-10-11T11:16:20.802 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~49s 2023-10-11T11:16:26.205 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~54s 2023-10-11T11:16:29.702 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi111.stderr:2023-10-11T11:16:29.701+0000 7f1e68fcb700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x556983bc5ba0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-10-11T11:16:29.703 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi111.stderr:2023-10-11T11:16:29.701+0000 7f1e6efd7700 -1 rbd::mirror::ImageReplayer: 0x556984a6c280 [3/b48a49ee-b365-4773-a4fc-2556af272547] operator(): start failed: (85) Interrupted system call should be restarted 2023-10-11T11:16:31.609 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~59s 2023-10-11T11:16:37.012 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~65s 2023-10-11T11:16:37.768 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2023-10-11T11:16:37.769 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2023-10-11T11:16:37.769 INFO:tasks.rbd_mirror_thrash:waiting for 0 secs before reviving daemons 2023-10-11T11:16:37.769 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2023-10-11T11:16:37.770 INFO:teuthology.orchestra.run:waiting for 600 2023-10-11T11:16:37.770 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi111.stderr:2023-10-11T11:16:37.769+0000 7f607dc0b700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 140199) UID: 1000 2023-10-11T11:16:42.416 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~70s 2023-10-11T11:16:43.871 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2023-10-11T11:16:43.872 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2023-10-11T11:16:43.872 DEBUG:teuthology.orchestra.run.smithi111:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2023-10-11T11:16:43.875 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2023-10-11T11:16:43.876 INFO:tasks.rbd_mirror_thrash:waiting for 82 secs before thrashing 2023-10-11T11:16:47.819 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~76s 2023-10-11T11:16:48.683 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi111.stderr:2023-10-11T11:16:48.682+0000 7f2c296d2700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55c1e9721000 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=380 2023-10-11T11:16:48.684 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi111.stderr:2023-10-11T11:16:48.682+0000 7f2c296d2700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55c1e9721000 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=384, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=9a182971-2e50-451f-a839-7c08d3b6157c, primary_snap_id=17c, last_copied_object_number=128, snap_seqs={380=18446744073709551614}] 2023-10-11T11:16:48.690 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:Mirroring disabled 2023-10-11T11:16:48.704 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_present cluster1 mirror image_7 deleted 2023-10-11T11:16:48.704 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:16:48.704 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:16:48.704 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_7 2023-10-11T11:16:48.704 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local state=deleted 2023-10-11T11:16:48.705 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image_id= 2023-10-11T11:16:48.705 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T11:16:48.705 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test -n '' 2023-10-11T11:16:48.705 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ get_image_id cluster1 mirror image_7 2023-10-11T11:16:48.705 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:16:48.705 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local pool=mirror 2023-10-11T11:16:48.706 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local image=image_7 2023-10-11T11:16:48.706 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2023-10-11T11:16:48.706 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-10-11T11:16:48.743 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd: error opening image image_7: (2) No such file or directory 2023-10-11T11:16:48.747 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image_id= 2023-10-11T11:16:48.747 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-10-11T11:16:48.747 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.1 2023-10-11T11:16:48.848 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test_image_present cluster1 mirror image_7 deleted '' 2023-10-11T11:16:48.849 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:16:48.849 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:16:48.849 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_7 2023-10-11T11:16:48.849 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local test_state=deleted 2023-10-11T11:16:48.849 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image_id= 2023-10-11T11:16:48.850 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_state=deleted 2023-10-11T11:16:48.850 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_image_id 2023-10-11T11:16:48.850 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ get_image_id cluster1 mirror image_7 2023-10-11T11:16:48.850 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:16:48.850 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local pool=mirror 2023-10-11T11:16:48.850 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local image=image_7 2023-10-11T11:16:48.851 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2023-10-11T11:16:48.851 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-10-11T11:16:48.883 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd: error opening image image_7: (2) No such file or directory 2023-10-11T11:16:48.887 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ current_image_id= 2023-10-11T11:16:48.887 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test -n '' 2023-10-11T11:16:48.887 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test deleted = deleted 2023-10-11T11:16:48.888 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:16:48.888 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ purge_snapshots cluster2 mirror image_7 2023-10-11T11:16:48.888 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:16:48.888 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:16:48.888 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_7 2023-10-11T11:16:48.889 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 snap purge mirror/image_7 2023-10-11T11:16:48.932 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_image_retry cluster2 mirror image_7 2023-10-11T11:16:48.932 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:16:48.932 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:16:48.932 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_7 2023-10-11T11:16:48.932 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0 1 2 4 8 16 32 2023-10-11T11:16:48.933 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0 2023-10-11T11:16:48.933 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_image cluster2 mirror image_7 2023-10-11T11:16:48.934 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:16:48.934 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:16:48.934 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_7 2023-10-11T11:16:48.934 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_7 2023-10-11T11:16:48.977 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 rm mirror/image_7 2023-10-11T11:16:49.194 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2023-10-11T11:16:49.197 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:16:49.198 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 10` 2023-10-11T11:16:49.198 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_8 2023-10-11T11:16:49.198 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ create_image cluster2 mirror image_8 512M 2023-10-11T11:16:49.198 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:16:49.198 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T11:16:49.199 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:16:49.199 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T11:16:49.199 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_8 2023-10-11T11:16:49.199 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T11:16:49.199 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local size=128 2023-10-11T11:16:49.199 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n 512M ']' 2023-10-11T11:16:49.200 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ size=512M 2023-10-11T11:16:49.200 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T11:16:49.200 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_8 2023-10-11T11:16:49.245 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ enable_mirror cluster2 mirror image_8 2023-10-11T11:16:49.245 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:16:49.245 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:16:49.246 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_8 2023-10-11T11:16:49.246 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local mode=snapshot 2023-10-11T11:16:49.246 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_8 snapshot 2023-10-11T11:16:49.696 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:Mirroring enabled 2023-10-11T11:16:49.705 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 info mirror/image_8 2023-10-11T11:16:49.741 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:rbd image 'image_8': 2023-10-11T11:16:49.742 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: size 512 MiB in 128 objects 2023-10-11T11:16:49.742 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: order 22 (4 MiB objects) 2023-10-11T11:16:49.742 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: snapshot_count: 1 2023-10-11T11:16:49.742 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: id: 1967e09e9b4f 2023-10-11T11:16:49.742 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: block_name_prefix: rbd_data.1967e09e9b4f 2023-10-11T11:16:49.743 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: format: 2 2023-10-11T11:16:49.743 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-10-11T11:16:49.743 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: op_features: 2023-10-11T11:16:49.743 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: flags: 2023-10-11T11:16:49.743 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: create_timestamp: Wed Oct 11 11:16:49 2023 2023-10-11T11:16:49.744 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: access_timestamp: Wed Oct 11 11:16:49 2023 2023-10-11T11:16:49.744 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: modify_timestamp: Wed Oct 11 11:16:49 2023 2023-10-11T11:16:49.744 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring state: enabled 2023-10-11T11:16:49.744 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring mode: snapshot 2023-10-11T11:16:49.744 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring global id: 95a1a3d7-9f1a-4cb6-8fab-c33c10a16228 2023-10-11T11:16:49.744 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring primary: true 2023-10-11T11:16:49.745 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ stress_write_image cluster2 mirror image_8 2023-10-11T11:16:49.745 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:16:49.746 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:16:49.746 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_8 2023-10-11T11:16:49.746 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-10-11T11:16:49.747 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local duration=5 2023-10-11T11:16:49.747 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ set +e 2023-10-11T11:16:49.747 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_8 --debug-rbd=20 --debug-journaler=20 2023-10-11T11:16:49.981 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:100 IOs, 29180928 bytes 2023-10-11T11:16:50.109 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:200 IOs, 51864576 bytes 2023-10-11T11:16:50.256 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:300 IOs, 95106048 bytes 2023-10-11T11:16:50.342 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:400 IOs, 126754816 bytes 2023-10-11T11:16:50.470 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:500 IOs, 154968064 bytes 2023-10-11T11:16:50.561 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:600 IOs, 162305024 bytes 2023-10-11T11:16:50.701 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:700 IOs, 214190080 bytes 2023-10-11T11:16:50.813 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:800 IOs, 237807616 bytes 2023-10-11T11:16:50.922 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:900 IOs, 274102272 bytes 2023-10-11T11:16:51.028 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1000 IOs, 309745664 bytes 2023-10-11T11:16:51.127 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1100 IOs, 316592128 bytes 2023-10-11T11:16:51.306 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1200 IOs, 389233664 bytes 2023-10-11T11:16:51.411 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1300 IOs, 445928448 bytes 2023-10-11T11:16:51.553 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1400 IOs, 472077312 bytes 2023-10-11T11:16:51.664 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1500 IOs, 503454720 bytes 2023-10-11T11:16:51.741 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1600 IOs, 504068096 bytes 2023-10-11T11:16:51.913 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1700 IOs, 589652992 bytes 2023-10-11T11:16:52.019 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1800 IOs, 617081856 bytes 2023-10-11T11:16:52.128 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1900 IOs, 648564736 bytes 2023-10-11T11:16:52.284 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2000 IOs, 696805376 bytes 2023-10-11T11:16:52.539 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2100 IOs, 846894080 bytes 2023-10-11T11:16:52.700 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2200 IOs, 897481728 bytes 2023-10-11T11:16:52.899 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2300 IOs, 989088768 bytes 2023-10-11T11:16:53.075 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2400 IOs, 1042957312 bytes 2023-10-11T11:16:53.209 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2500 IOs, 1081257984 bytes 2023-10-11T11:16:53.224 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~81s 2023-10-11T11:16:53.319 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2600 IOs, 1117532160 bytes 2023-10-11T11:16:53.453 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2700 IOs, 1148043264 bytes 2023-10-11T11:16:53.590 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2800 IOs, 1194155008 bytes 2023-10-11T11:16:53.702 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2900 IOs, 1221226496 bytes 2023-10-11T11:16:53.818 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3000 IOs, 1248257024 bytes 2023-10-11T11:16:53.875 INFO:tasks.ceph.cluster2.osd.2.smithi111.stderr:2023-10-11T11:16:53.874+0000 7fa35bd9b700 -1 reset not still connected to 0x55643b40e270 2023-10-11T11:16:53.989 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3100 IOs, 1310909440 bytes 2023-10-11T11:16:54.126 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3200 IOs, 1355150336 bytes 2023-10-11T11:16:54.255 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3300 IOs, 1388807168 bytes 2023-10-11T11:16:54.384 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3400 IOs, 1426706432 bytes 2023-10-11T11:16:54.507 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3500 IOs, 1463588864 bytes 2023-10-11T11:16:54.682 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3600 IOs, 1542364160 bytes 2023-10-11T11:16:54.752 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ error_code=124 2023-10-11T11:16:54.752 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ set -e 2023-10-11T11:16:54.752 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' 124 -eq 124 ']' 2023-10-11T11:16:54.753 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:16:54.753 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_present cluster1 mirror image_8 present 2023-10-11T11:16:54.753 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:16:54.753 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:16:54.754 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_8 2023-10-11T11:16:54.754 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local state=present 2023-10-11T11:16:54.754 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image_id= 2023-10-11T11:16:54.754 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T11:16:54.755 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test -n '' 2023-10-11T11:16:54.755 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ get_image_id cluster1 mirror image_8 2023-10-11T11:16:54.756 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:16:54.756 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local pool=mirror 2023-10-11T11:16:54.756 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local image=image_8 2023-10-11T11:16:54.757 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2023-10-11T11:16:54.757 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-10-11T11:16:54.795 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image_id=131620929f11 2023-10-11T11:16:54.796 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-10-11T11:16:54.796 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.1 2023-10-11T11:16:54.897 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test_image_present cluster1 mirror image_8 present 131620929f11 2023-10-11T11:16:54.897 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:16:54.898 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:16:54.898 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_8 2023-10-11T11:16:54.898 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local test_state=present 2023-10-11T11:16:54.899 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image_id=131620929f11 2023-10-11T11:16:54.899 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_state=deleted 2023-10-11T11:16:54.899 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_image_id 2023-10-11T11:16:54.899 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ get_image_id cluster1 mirror image_8 2023-10-11T11:16:54.900 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:16:54.900 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local pool=mirror 2023-10-11T11:16:54.900 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local image=image_8 2023-10-11T11:16:54.900 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2023-10-11T11:16:54.901 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-10-11T11:16:54.942 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ current_image_id=131620929f11 2023-10-11T11:16:54.942 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test -n 131620929f11 2023-10-11T11:16:54.942 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test -z 131620929f11 -o 131620929f11 = 131620929f11 2023-10-11T11:16:54.943 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ current_state=present 2023-10-11T11:16:54.943 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test present = present 2023-10-11T11:16:54.943 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:16:54.943 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ disable_mirror cluster2 mirror image_8 2023-10-11T11:16:54.944 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:16:54.944 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:16:54.944 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_8 2023-10-11T11:16:54.944 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_8 2023-10-11T11:16:58.628 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~86s 2023-10-11T11:16:59.702 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi111.stderr:2023-10-11T11:16:59.701+0000 7f1e687ca700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x556983ebad00 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-10-11T11:16:59.703 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi111.stderr:2023-10-11T11:16:59.701+0000 7f1e6f7d8700 -1 rbd::mirror::ImageReplayer: 0x5569846da780 [3/95a1a3d7-9f1a-4cb6-8fab-c33c10a16228] operator(): start failed: (85) Interrupted system call should be restarted 2023-10-11T11:17:04.032 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~92s 2023-10-11T11:17:09.436 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~97s 2023-10-11T11:17:14.838 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~103s 2023-10-11T11:17:20.242 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~108s 2023-10-11T11:17:25.645 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~113s 2023-10-11T11:17:25.686 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi111.stderr:2023-10-11T11:17:25.685+0000 7f2c29ed3700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55c1e94f7800 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=382 2023-10-11T11:17:25.686 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi111.stderr:2023-10-11T11:17:25.685+0000 7f2c29ed3700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55c1e94f7800 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=386, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=9a182971-2e50-451f-a839-7c08d3b6157c, primary_snap_id=17e, last_copied_object_number=128, snap_seqs={382=18446744073709551614}] 2023-10-11T11:17:25.690 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:Mirroring disabled 2023-10-11T11:17:25.705 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_present cluster1 mirror image_8 deleted 2023-10-11T11:17:25.705 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:17:25.705 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:17:25.705 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_8 2023-10-11T11:17:25.706 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local state=deleted 2023-10-11T11:17:25.706 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image_id= 2023-10-11T11:17:25.706 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T11:17:25.706 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test -n '' 2023-10-11T11:17:25.707 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ get_image_id cluster1 mirror image_8 2023-10-11T11:17:25.707 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:17:25.707 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local pool=mirror 2023-10-11T11:17:25.707 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local image=image_8 2023-10-11T11:17:25.707 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2023-10-11T11:17:25.708 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-10-11T11:17:25.741 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd: error opening image image_8: (2) No such file or directory 2023-10-11T11:17:25.745 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image_id= 2023-10-11T11:17:25.745 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-10-11T11:17:25.745 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.1 2023-10-11T11:17:25.847 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test_image_present cluster1 mirror image_8 deleted '' 2023-10-11T11:17:25.847 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:17:25.848 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:17:25.848 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_8 2023-10-11T11:17:25.848 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local test_state=deleted 2023-10-11T11:17:25.848 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image_id= 2023-10-11T11:17:25.848 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_state=deleted 2023-10-11T11:17:25.849 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_image_id 2023-10-11T11:17:25.849 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ get_image_id cluster1 mirror image_8 2023-10-11T11:17:25.849 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:17:25.849 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local pool=mirror 2023-10-11T11:17:25.849 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local image=image_8 2023-10-11T11:17:25.850 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2023-10-11T11:17:25.850 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-10-11T11:17:25.881 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd: error opening image image_8: (2) No such file or directory 2023-10-11T11:17:25.884 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ current_image_id= 2023-10-11T11:17:25.884 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test -n '' 2023-10-11T11:17:25.885 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test deleted = deleted 2023-10-11T11:17:25.885 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:17:25.885 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ purge_snapshots cluster2 mirror image_8 2023-10-11T11:17:25.885 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:17:25.885 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:17:25.886 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_8 2023-10-11T11:17:25.886 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 snap purge mirror/image_8 2023-10-11T11:17:25.928 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_image_retry cluster2 mirror image_8 2023-10-11T11:17:25.929 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:17:25.929 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:17:25.929 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_8 2023-10-11T11:17:25.929 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0 1 2 4 8 16 32 2023-10-11T11:17:25.929 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0 2023-10-11T11:17:25.930 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_image cluster2 mirror image_8 2023-10-11T11:17:25.930 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:17:25.931 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:17:25.931 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_8 2023-10-11T11:17:25.931 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_8 2023-10-11T11:17:25.973 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 rm mirror/image_8 2023-10-11T11:17:26.201 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2023-10-11T11:17:26.205 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:17:26.205 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 10` 2023-10-11T11:17:26.205 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_9 2023-10-11T11:17:26.205 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ create_image cluster2 mirror image_9 512M 2023-10-11T11:17:26.205 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:17:26.206 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T11:17:26.206 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:17:26.206 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T11:17:26.206 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_9 2023-10-11T11:17:26.207 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T11:17:26.207 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local size=128 2023-10-11T11:17:26.207 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n 512M ']' 2023-10-11T11:17:26.207 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ size=512M 2023-10-11T11:17:26.208 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T11:17:26.208 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_9 2023-10-11T11:17:26.253 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ enable_mirror cluster2 mirror image_9 2023-10-11T11:17:26.253 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:17:26.253 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:17:26.253 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_9 2023-10-11T11:17:26.254 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local mode=snapshot 2023-10-11T11:17:26.254 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_9 snapshot 2023-10-11T11:17:26.698 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:Mirroring enabled 2023-10-11T11:17:26.707 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 info mirror/image_9 2023-10-11T11:17:26.745 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:rbd image 'image_9': 2023-10-11T11:17:26.745 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: size 512 MiB in 128 objects 2023-10-11T11:17:26.745 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: order 22 (4 MiB objects) 2023-10-11T11:17:26.745 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: snapshot_count: 1 2023-10-11T11:17:26.746 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: id: 197f12e3409b 2023-10-11T11:17:26.746 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: block_name_prefix: rbd_data.197f12e3409b 2023-10-11T11:17:26.746 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: format: 2 2023-10-11T11:17:26.746 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-10-11T11:17:26.746 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: op_features: 2023-10-11T11:17:26.747 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: flags: 2023-10-11T11:17:26.747 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: create_timestamp: Wed Oct 11 11:17:26 2023 2023-10-11T11:17:26.747 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: access_timestamp: Wed Oct 11 11:17:26 2023 2023-10-11T11:17:26.747 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: modify_timestamp: Wed Oct 11 11:17:26 2023 2023-10-11T11:17:26.748 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring state: enabled 2023-10-11T11:17:26.748 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring mode: snapshot 2023-10-11T11:17:26.748 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring global id: 6b3a4a83-c120-4c5a-9d6b-2c1aa3fe7bde 2023-10-11T11:17:26.748 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring primary: true 2023-10-11T11:17:26.749 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ stress_write_image cluster2 mirror image_9 2023-10-11T11:17:26.750 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:17:26.750 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:17:26.750 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_9 2023-10-11T11:17:26.750 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-10-11T11:17:26.751 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local duration=14 2023-10-11T11:17:26.751 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ set +e 2023-10-11T11:17:26.751 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_9 --debug-rbd=20 --debug-journaler=20 2023-10-11T11:17:26.970 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:100 IOs, 43070464 bytes 2023-10-11T11:17:27.098 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:200 IOs, 60718080 bytes 2023-10-11T11:17:27.256 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:300 IOs, 146982912 bytes 2023-10-11T11:17:27.421 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:400 IOs, 199770112 bytes 2023-10-11T11:17:27.509 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:500 IOs, 203060224 bytes 2023-10-11T11:17:27.629 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:600 IOs, 238798848 bytes 2023-10-11T11:17:27.777 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:700 IOs, 306842624 bytes 2023-10-11T11:17:27.966 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:800 IOs, 399060992 bytes 2023-10-11T11:17:28.082 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:900 IOs, 435649536 bytes 2023-10-11T11:17:28.272 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1000 IOs, 514435072 bytes 2023-10-11T11:17:28.407 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1100 IOs, 583108608 bytes 2023-10-11T11:17:28.574 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1200 IOs, 659589120 bytes 2023-10-11T11:17:28.696 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1300 IOs, 682816512 bytes 2023-10-11T11:17:28.941 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1400 IOs, 817701888 bytes 2023-10-11T11:17:29.088 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1500 IOs, 885433344 bytes 2023-10-11T11:17:29.170 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1600 IOs, 891820032 bytes 2023-10-11T11:17:29.250 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1700 IOs, 903312384 bytes 2023-10-11T11:17:29.353 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1800 IOs, 938629120 bytes 2023-10-11T11:17:29.532 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1900 IOs, 1010252800 bytes 2023-10-11T11:17:29.695 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2000 IOs, 1064784896 bytes 2023-10-11T11:17:29.780 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2100 IOs, 1097729024 bytes 2023-10-11T11:17:29.924 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2200 IOs, 1125493760 bytes 2023-10-11T11:17:30.042 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2300 IOs, 1158050816 bytes 2023-10-11T11:17:30.133 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2400 IOs, 1185319936 bytes 2023-10-11T11:17:30.259 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2500 IOs, 1205325824 bytes 2023-10-11T11:17:30.428 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2600 IOs, 1263947776 bytes 2023-10-11T11:17:30.565 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2700 IOs, 1319357440 bytes 2023-10-11T11:17:30.697 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2800 IOs, 1366545408 bytes 2023-10-11T11:17:30.779 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2900 IOs, 1367168000 bytes 2023-10-11T11:17:30.942 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3000 IOs, 1440198656 bytes 2023-10-11T11:17:31.048 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~119s 2023-10-11T11:17:31.058 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3100 IOs, 1467930624 bytes 2023-10-11T11:17:31.191 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3200 IOs, 1524636672 bytes 2023-10-11T11:17:31.270 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3300 IOs, 1528181760 bytes 2023-10-11T11:17:31.275 INFO:tasks.ceph.cluster2.osd.1.smithi111.stderr:2023-10-11T11:17:31.274+0000 7f9628714700 -1 reset not still connected to 0x56427692add0 2023-10-11T11:17:31.350 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3400 IOs, 1532649472 bytes 2023-10-11T11:17:31.487 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3500 IOs, 1590214656 bytes 2023-10-11T11:17:31.597 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3600 IOs, 1612158976 bytes 2023-10-11T11:17:31.682 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3700 IOs, 1617344512 bytes 2023-10-11T11:17:31.843 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3800 IOs, 1692085248 bytes 2023-10-11T11:17:31.963 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3900 IOs, 1718689792 bytes 2023-10-11T11:17:32.138 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4000 IOs, 1785724928 bytes 2023-10-11T11:17:32.285 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4100 IOs, 1840836608 bytes 2023-10-11T11:17:32.421 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4200 IOs, 1895678976 bytes 2023-10-11T11:17:32.525 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4300 IOs, 1939023872 bytes 2023-10-11T11:17:32.622 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4400 IOs, 1950724096 bytes 2023-10-11T11:17:32.770 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4500 IOs, 2010063872 bytes 2023-10-11T11:17:32.893 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4600 IOs, 2059327488 bytes 2023-10-11T11:17:32.974 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4700 IOs, 2077381632 bytes 2023-10-11T11:17:33.085 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4800 IOs, 2106222592 bytes 2023-10-11T11:17:33.162 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4900 IOs, 2108794880 bytes 2023-10-11T11:17:33.338 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5000 IOs, 2178527232 bytes 2023-10-11T11:17:33.413 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5100 IOs, 2179123200 bytes 2023-10-11T11:17:33.605 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5200 IOs, 2282259456 bytes 2023-10-11T11:17:33.751 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5300 IOs, 2332682240 bytes 2023-10-11T11:17:33.856 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5400 IOs, 2347767808 bytes 2023-10-11T11:17:34.037 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5500 IOs, 2427478016 bytes 2023-10-11T11:17:34.163 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5600 IOs, 2479808512 bytes 2023-10-11T11:17:34.338 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5700 IOs, 2576432128 bytes 2023-10-11T11:17:34.514 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5800 IOs, 2647919616 bytes 2023-10-11T11:17:34.686 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5900 IOs, 2707520512 bytes 2023-10-11T11:17:34.793 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6000 IOs, 2733986816 bytes 2023-10-11T11:17:34.901 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6100 IOs, 2752206848 bytes 2023-10-11T11:17:34.974 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6200 IOs, 2761973760 bytes 2023-10-11T11:17:35.085 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6300 IOs, 2802613248 bytes 2023-10-11T11:17:35.259 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6400 IOs, 2869281792 bytes 2023-10-11T11:17:35.392 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6500 IOs, 2910139392 bytes 2023-10-11T11:17:35.561 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6600 IOs, 2996036608 bytes 2023-10-11T11:17:35.606 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.42272646217577037) > weight (0.25) 2023-10-11T11:17:35.606 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.555159165694095) > weight (0.25) 2023-10-11T11:17:35.606 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.6170937547599544) > weight (0.25) 2023-10-11T11:17:35.606 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.3531062397520245) > weight (0.25) 2023-10-11T11:17:35.606 INFO:tasks.rbd_mirror_thrash:waiting for 90 secs before thrashing 2023-10-11T11:17:35.738 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6700 IOs, 3061985280 bytes 2023-10-11T11:17:35.904 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6800 IOs, 3137668096 bytes 2023-10-11T11:17:36.033 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6900 IOs, 3178805248 bytes 2023-10-11T11:17:36.138 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:7000 IOs, 3210225664 bytes 2023-10-11T11:17:36.276 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:7100 IOs, 3248922624 bytes 2023-10-11T11:17:36.355 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:7200 IOs, 3253219328 bytes 2023-10-11T11:17:36.451 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~124s 2023-10-11T11:17:36.507 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:7300 IOs, 3313381376 bytes 2023-10-11T11:17:36.660 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:7400 IOs, 3368488960 bytes 2023-10-11T11:17:36.748 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:7500 IOs, 3381343232 bytes 2023-10-11T11:17:36.867 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:7600 IOs, 3419273216 bytes 2023-10-11T11:17:36.997 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:7700 IOs, 3464901632 bytes 2023-10-11T11:17:37.124 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:7800 IOs, 3503502336 bytes 2023-10-11T11:17:37.231 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:7900 IOs, 3525066752 bytes 2023-10-11T11:17:37.455 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:8000 IOs, 3626180608 bytes 2023-10-11T11:17:37.604 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:8100 IOs, 3689772032 bytes 2023-10-11T11:17:37.704 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:8200 IOs, 3702790144 bytes 2023-10-11T11:17:37.858 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:8300 IOs, 3761655808 bytes 2023-10-11T11:17:38.029 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:8400 IOs, 3863790592 bytes 2023-10-11T11:17:38.191 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:8500 IOs, 3912323072 bytes 2023-10-11T11:17:38.322 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:8600 IOs, 3954250752 bytes 2023-10-11T11:17:38.436 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:8700 IOs, 4007869440 bytes 2023-10-11T11:17:38.623 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:8800 IOs, 4063302656 bytes 2023-10-11T11:17:38.807 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:8900 IOs, 4148775936 bytes 2023-10-11T11:17:38.949 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:9000 IOs, 4226608128 bytes 2023-10-11T11:17:39.056 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:9100 IOs, 4238020608 bytes 2023-10-11T11:17:39.179 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:9200 IOs, 4282247168 bytes 2023-10-11T11:17:39.267 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:9300 IOs, 4291686400 bytes 2023-10-11T11:17:39.445 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:9400 IOs, 4366856192 bytes 2023-10-11T11:17:39.587 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:9500 IOs, 4421664768 bytes 2023-10-11T11:17:39.697 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:9600 IOs, 4452651008 bytes 2023-10-11T11:17:39.823 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:9700 IOs, 4500924416 bytes 2023-10-11T11:17:39.977 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:9800 IOs, 4549737472 bytes 2023-10-11T11:17:40.095 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:9900 IOs, 4587268096 bytes 2023-10-11T11:17:40.256 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:10000 IOs, 4660731904 bytes 2023-10-11T11:17:40.344 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:10100 IOs, 4674270208 bytes 2023-10-11T11:17:40.524 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:10200 IOs, 4741542912 bytes 2023-10-11T11:17:40.699 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:10300 IOs, 4833150976 bytes 2023-10-11T11:17:40.754 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ error_code=124 2023-10-11T11:17:40.754 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ set -e 2023-10-11T11:17:40.754 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' 124 -eq 124 ']' 2023-10-11T11:17:40.755 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:17:40.755 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_present cluster1 mirror image_9 present 2023-10-11T11:17:40.755 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:17:40.755 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:17:40.755 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_9 2023-10-11T11:17:40.756 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local state=present 2023-10-11T11:17:40.756 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image_id= 2023-10-11T11:17:40.756 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T11:17:40.756 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test -n '' 2023-10-11T11:17:40.756 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ get_image_id cluster1 mirror image_9 2023-10-11T11:17:40.757 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:17:40.757 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local pool=mirror 2023-10-11T11:17:40.757 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local image=image_9 2023-10-11T11:17:40.757 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2023-10-11T11:17:40.758 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-10-11T11:17:40.797 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image_id=1316e78452cf 2023-10-11T11:17:40.798 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-10-11T11:17:40.798 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.1 2023-10-11T11:17:40.899 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test_image_present cluster1 mirror image_9 present 1316e78452cf 2023-10-11T11:17:40.899 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:17:40.899 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:17:40.900 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_9 2023-10-11T11:17:40.900 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local test_state=present 2023-10-11T11:17:40.900 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image_id=1316e78452cf 2023-10-11T11:17:40.900 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_state=deleted 2023-10-11T11:17:40.900 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_image_id 2023-10-11T11:17:40.901 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ get_image_id cluster1 mirror image_9 2023-10-11T11:17:40.901 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:17:40.901 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local pool=mirror 2023-10-11T11:17:40.901 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local image=image_9 2023-10-11T11:17:40.901 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2023-10-11T11:17:40.901 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-10-11T11:17:40.942 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ current_image_id=1316e78452cf 2023-10-11T11:17:40.943 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test -n 1316e78452cf 2023-10-11T11:17:40.943 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test -z 1316e78452cf -o 1316e78452cf = 1316e78452cf 2023-10-11T11:17:40.943 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ current_state=present 2023-10-11T11:17:40.944 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test present = present 2023-10-11T11:17:40.944 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:17:40.944 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ disable_mirror cluster2 mirror image_9 2023-10-11T11:17:40.944 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:17:40.944 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:17:40.945 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_9 2023-10-11T11:17:40.945 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_9 2023-10-11T11:17:41.855 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~130s 2023-10-11T11:17:47.259 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~135s 2023-10-11T11:17:52.662 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~140s 2023-10-11T11:17:58.066 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~146s 2023-10-11T11:17:59.703 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi111.stderr:2023-10-11T11:17:59.701+0000 7f1e687ca700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x556983ebab60 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-10-11T11:17:59.703 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi111.stderr:2023-10-11T11:17:59.701+0000 7f1e6f7d8700 -1 rbd::mirror::ImageReplayer: 0x556984a6c280 [3/6b3a4a83-c120-4c5a-9d6b-2c1aa3fe7bde] operator(): start failed: (85) Interrupted system call should be restarted 2023-10-11T11:18:03.470 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~151s 2023-10-11T11:18:05.877 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2023-10-11T11:18:05.878 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2023-10-11T11:18:05.878 INFO:tasks.rbd_mirror_thrash:waiting for 3 secs before reviving daemons 2023-10-11T11:18:05.879 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi111.stderr:2023-10-11T11:18:05.878+0000 7fe3b71b7700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 141835) UID: 1000 2023-10-11T11:18:08.874 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~157s 2023-10-11T11:18:08.880 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2023-10-11T11:18:08.880 INFO:teuthology.orchestra.run:waiting for 600 2023-10-11T11:18:08.880 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2023-10-11T11:18:08.881 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2023-10-11T11:18:08.881 DEBUG:teuthology.orchestra.run.smithi111:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2023-10-11T11:18:08.884 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2023-10-11T11:18:08.884 INFO:tasks.rbd_mirror_thrash:waiting for 115 secs before thrashing 2023-10-11T11:18:11.710 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi111.stderr:2023-10-11T11:18:11.709+0000 7f2c29ed3700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55c1e87da400 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=384 2023-10-11T11:18:11.710 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi111.stderr:2023-10-11T11:18:11.709+0000 7f2c29ed3700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55c1e87da400 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=388, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=9a182971-2e50-451f-a839-7c08d3b6157c, primary_snap_id=180, last_copied_object_number=128, snap_seqs={384=18446744073709551614}] 2023-10-11T11:18:11.718 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:Mirroring disabled 2023-10-11T11:18:11.730 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_present cluster1 mirror image_9 deleted 2023-10-11T11:18:11.730 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:18:11.730 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:18:11.731 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_9 2023-10-11T11:18:11.731 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local state=deleted 2023-10-11T11:18:11.731 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image_id= 2023-10-11T11:18:11.731 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T11:18:11.731 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test -n '' 2023-10-11T11:18:11.732 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ get_image_id cluster1 mirror image_9 2023-10-11T11:18:11.732 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:18:11.732 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local pool=mirror 2023-10-11T11:18:11.732 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local image=image_9 2023-10-11T11:18:11.732 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2023-10-11T11:18:11.733 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-10-11T11:18:11.766 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd: error opening image image_9: (2) No such file or directory 2023-10-11T11:18:11.769 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image_id= 2023-10-11T11:18:11.769 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-10-11T11:18:11.770 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.1 2023-10-11T11:18:11.871 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test_image_present cluster1 mirror image_9 deleted '' 2023-10-11T11:18:11.871 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:18:11.872 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:18:11.872 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_9 2023-10-11T11:18:11.872 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local test_state=deleted 2023-10-11T11:18:11.872 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image_id= 2023-10-11T11:18:11.872 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_state=deleted 2023-10-11T11:18:11.873 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_image_id 2023-10-11T11:18:11.873 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ get_image_id cluster1 mirror image_9 2023-10-11T11:18:11.873 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:18:11.873 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local pool=mirror 2023-10-11T11:18:11.873 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local image=image_9 2023-10-11T11:18:11.874 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-10-11T11:18:11.874 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2023-10-11T11:18:11.909 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd: error opening image image_9: (2) No such file or directory 2023-10-11T11:18:11.912 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ current_image_id= 2023-10-11T11:18:11.913 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test -n '' 2023-10-11T11:18:11.913 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test deleted = deleted 2023-10-11T11:18:11.913 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:18:11.913 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ purge_snapshots cluster2 mirror image_9 2023-10-11T11:18:11.913 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:18:11.914 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:18:11.914 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_9 2023-10-11T11:18:11.914 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 snap purge mirror/image_9 2023-10-11T11:18:11.955 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_image_retry cluster2 mirror image_9 2023-10-11T11:18:11.955 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:18:11.956 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:18:11.956 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_9 2023-10-11T11:18:11.956 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0 1 2 4 8 16 32 2023-10-11T11:18:11.956 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0 2023-10-11T11:18:11.957 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_image cluster2 mirror image_9 2023-10-11T11:18:11.957 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:18:11.958 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:18:11.958 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_9 2023-10-11T11:18:11.958 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_9 2023-10-11T11:18:11.999 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 rm mirror/image_9 2023-10-11T11:18:12.217 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2023-10-11T11:18:12.221 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:18:12.221 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for i in `seq 1 10` 2023-10-11T11:18:12.221 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image=image_10 2023-10-11T11:18:12.221 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ create_image cluster2 mirror image_10 512M 2023-10-11T11:18:12.222 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:18:12.222 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T11:18:12.222 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:18:12.222 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T11:18:12.222 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_10 2023-10-11T11:18:12.223 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T11:18:12.223 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local size=128 2023-10-11T11:18:12.223 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -n 512M ']' 2023-10-11T11:18:12.223 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ size=512M 2023-10-11T11:18:12.223 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ shift 2023-10-11T11:18:12.224 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_10 2023-10-11T11:18:12.269 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ enable_mirror cluster2 mirror image_10 2023-10-11T11:18:12.269 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:18:12.269 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:18:12.270 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_10 2023-10-11T11:18:12.270 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local mode=snapshot 2023-10-11T11:18:12.270 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_10 snapshot 2023-10-11T11:18:12.725 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:Mirroring enabled 2023-10-11T11:18:12.735 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 info mirror/image_10 2023-10-11T11:18:12.772 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:rbd image 'image_10': 2023-10-11T11:18:12.772 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: size 512 MiB in 128 objects 2023-10-11T11:18:12.772 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: order 22 (4 MiB objects) 2023-10-11T11:18:12.772 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: snapshot_count: 1 2023-10-11T11:18:12.773 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: id: 199db72168b 2023-10-11T11:18:12.773 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: block_name_prefix: rbd_data.199db72168b 2023-10-11T11:18:12.773 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: format: 2 2023-10-11T11:18:12.773 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-10-11T11:18:12.774 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: op_features: 2023-10-11T11:18:12.774 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: flags: 2023-10-11T11:18:12.774 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: create_timestamp: Wed Oct 11 11:18:12 2023 2023-10-11T11:18:12.774 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: access_timestamp: Wed Oct 11 11:18:12 2023 2023-10-11T11:18:12.774 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: modify_timestamp: Wed Oct 11 11:18:12 2023 2023-10-11T11:18:12.775 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring state: enabled 2023-10-11T11:18:12.775 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring mode: snapshot 2023-10-11T11:18:12.775 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring global id: 8bca97e8-61d3-4c2b-aea1-515700fbe799 2023-10-11T11:18:12.775 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout: mirroring primary: true 2023-10-11T11:18:12.776 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ stress_write_image cluster2 mirror image_10 2023-10-11T11:18:12.776 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:18:12.776 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:18:12.777 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_10 2023-10-11T11:18:12.777 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-10-11T11:18:12.777 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local duration=11 2023-10-11T11:18:12.778 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ set +e 2023-10-11T11:18:12.778 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_10 --debug-rbd=20 --debug-journaler=20 2023-10-11T11:18:13.035 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:100 IOs, 50338816 bytes 2023-10-11T11:18:13.197 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:200 IOs, 104631296 bytes 2023-10-11T11:18:13.376 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:300 IOs, 186655744 bytes 2023-10-11T11:18:13.499 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:400 IOs, 204621824 bytes 2023-10-11T11:18:13.633 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:500 IOs, 256705536 bytes 2023-10-11T11:18:13.752 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:600 IOs, 300441600 bytes 2023-10-11T11:18:13.940 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:700 IOs, 359294976 bytes 2023-10-11T11:18:14.062 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:800 IOs, 403430400 bytes 2023-10-11T11:18:14.190 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:900 IOs, 463179776 bytes 2023-10-11T11:18:14.278 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~162s 2023-10-11T11:18:14.300 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1000 IOs, 483969024 bytes 2023-10-11T11:18:14.447 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1100 IOs, 554061824 bytes 2023-10-11T11:18:14.617 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1200 IOs, 601923584 bytes 2023-10-11T11:18:14.758 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1300 IOs, 668372992 bytes 2023-10-11T11:18:14.876 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1400 IOs, 715276288 bytes 2023-10-11T11:18:15.048 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1500 IOs, 766359552 bytes 2023-10-11T11:18:15.175 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1600 IOs, 823872512 bytes 2023-10-11T11:18:15.300 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1700 IOs, 846517248 bytes 2023-10-11T11:18:15.445 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1800 IOs, 901602304 bytes 2023-10-11T11:18:15.553 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:1900 IOs, 925996032 bytes 2023-10-11T11:18:15.737 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2000 IOs, 1007659008 bytes 2023-10-11T11:18:15.918 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2100 IOs, 1088181248 bytes 2023-10-11T11:18:16.031 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2200 IOs, 1121786880 bytes 2023-10-11T11:18:16.110 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2300 IOs, 1122373632 bytes 2023-10-11T11:18:16.194 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2400 IOs, 1129447424 bytes 2023-10-11T11:18:16.292 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2500 IOs, 1148487680 bytes 2023-10-11T11:18:16.374 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2600 IOs, 1151932416 bytes 2023-10-11T11:18:16.490 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2700 IOs, 1178335232 bytes 2023-10-11T11:18:16.624 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2800 IOs, 1220588544 bytes 2023-10-11T11:18:16.765 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:2900 IOs, 1287447552 bytes 2023-10-11T11:18:16.889 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3000 IOs, 1328855040 bytes 2023-10-11T11:18:16.993 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3100 IOs, 1358080000 bytes 2023-10-11T11:18:17.082 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3200 IOs, 1371087872 bytes 2023-10-11T11:18:17.168 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3300 IOs, 1400263680 bytes 2023-10-11T11:18:17.315 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3400 IOs, 1448495104 bytes 2023-10-11T11:18:17.483 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3500 IOs, 1526439936 bytes 2023-10-11T11:18:17.599 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3600 IOs, 1555614720 bytes 2023-10-11T11:18:17.743 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3700 IOs, 1618149376 bytes 2023-10-11T11:18:17.867 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3800 IOs, 1655016448 bytes 2023-10-11T11:18:17.944 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:3900 IOs, 1656658944 bytes 2023-10-11T11:18:18.042 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4000 IOs, 1682607104 bytes 2023-10-11T11:18:18.153 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4100 IOs, 1723924480 bytes 2023-10-11T11:18:18.273 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4200 IOs, 1746866176 bytes 2023-10-11T11:18:18.363 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4300 IOs, 1751658496 bytes 2023-10-11T11:18:18.498 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4400 IOs, 1799883776 bytes 2023-10-11T11:18:18.669 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4500 IOs, 1857503232 bytes 2023-10-11T11:18:18.807 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4600 IOs, 1907195904 bytes 2023-10-11T11:18:18.912 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4700 IOs, 1936116736 bytes 2023-10-11T11:18:19.043 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4800 IOs, 1982065664 bytes 2023-10-11T11:18:19.154 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:4900 IOs, 2016381952 bytes 2023-10-11T11:18:19.277 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5000 IOs, 2062916608 bytes 2023-10-11T11:18:19.386 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5100 IOs, 2083956736 bytes 2023-10-11T11:18:19.475 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5200 IOs, 2091243520 bytes 2023-10-11T11:18:19.569 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5300 IOs, 2091847680 bytes 2023-10-11T11:18:19.682 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~168s 2023-10-11T11:18:19.742 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5400 IOs, 2172302336 bytes 2023-10-11T11:18:19.914 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5500 IOs, 2235110400 bytes 2023-10-11T11:18:20.037 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5600 IOs, 2275804160 bytes 2023-10-11T11:18:20.113 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5700 IOs, 2284899328 bytes 2023-10-11T11:18:20.297 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5800 IOs, 2403088384 bytes 2023-10-11T11:18:20.623 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:5900 IOs, 2509043712 bytes 2023-10-11T11:18:20.743 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6000 IOs, 2549280768 bytes 2023-10-11T11:18:20.892 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6100 IOs, 2620775424 bytes 2023-10-11T11:18:21.024 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6200 IOs, 2666090496 bytes 2023-10-11T11:18:21.183 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6300 IOs, 2736144384 bytes 2023-10-11T11:18:21.296 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6400 IOs, 2766038016 bytes 2023-10-11T11:18:21.453 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6500 IOs, 2823422976 bytes 2023-10-11T11:18:21.573 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6600 IOs, 2862561280 bytes 2023-10-11T11:18:21.693 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6700 IOs, 2895472640 bytes 2023-10-11T11:18:21.837 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6800 IOs, 2969749504 bytes 2023-10-11T11:18:21.938 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:6900 IOs, 2978022400 bytes 2023-10-11T11:18:22.043 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:7000 IOs, 2986281984 bytes 2023-10-11T11:18:22.140 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:7100 IOs, 3022660608 bytes 2023-10-11T11:18:22.312 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:7200 IOs, 3094010880 bytes 2023-10-11T11:18:22.482 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:7300 IOs, 3170256896 bytes 2023-10-11T11:18:22.667 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:7400 IOs, 3262270464 bytes 2023-10-11T11:18:22.817 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:7500 IOs, 3311116288 bytes 2023-10-11T11:18:22.891 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:7600 IOs, 3311742976 bytes 2023-10-11T11:18:23.022 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:7700 IOs, 3347899392 bytes 2023-10-11T11:18:23.119 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:7800 IOs, 3369348096 bytes 2023-10-11T11:18:23.257 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:7900 IOs, 3420293120 bytes 2023-10-11T11:18:23.413 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:8000 IOs, 3483819008 bytes 2023-10-11T11:18:23.597 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:8100 IOs, 3553616896 bytes 2023-10-11T11:18:23.707 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:8200 IOs, 3589955584 bytes 2023-10-11T11:18:23.785 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ error_code=124 2023-10-11T11:18:23.786 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ set -e 2023-10-11T11:18:23.786 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' 124 -eq 124 ']' 2023-10-11T11:18:23.786 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:18:23.786 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_present cluster1 mirror image_10 present 2023-10-11T11:18:23.786 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:18:23.787 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:18:23.787 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_10 2023-10-11T11:18:23.787 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local state=present 2023-10-11T11:18:23.787 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image_id= 2023-10-11T11:18:23.787 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T11:18:23.788 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test -n '' 2023-10-11T11:18:23.788 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ get_image_id cluster1 mirror image_10 2023-10-11T11:18:23.788 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:18:23.789 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local pool=mirror 2023-10-11T11:18:23.789 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local image=image_10 2023-10-11T11:18:23.789 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ rbd --cluster=cluster1 info mirror/image_10 2023-10-11T11:18:23.789 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-10-11T11:18:23.830 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image_id=131615c354d0 2023-10-11T11:18:23.830 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-10-11T11:18:23.830 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.1 2023-10-11T11:18:23.932 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test_image_present cluster1 mirror image_10 present 131615c354d0 2023-10-11T11:18:23.932 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:18:23.932 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:18:23.932 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_10 2023-10-11T11:18:23.933 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local test_state=present 2023-10-11T11:18:23.933 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image_id=131615c354d0 2023-10-11T11:18:23.933 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_state=deleted 2023-10-11T11:18:23.933 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_image_id 2023-10-11T11:18:23.933 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ get_image_id cluster1 mirror image_10 2023-10-11T11:18:23.934 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:18:23.934 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local pool=mirror 2023-10-11T11:18:23.934 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local image=image_10 2023-10-11T11:18:23.934 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ rbd --cluster=cluster1 info mirror/image_10 2023-10-11T11:18:23.934 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-10-11T11:18:23.974 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ current_image_id=131615c354d0 2023-10-11T11:18:23.974 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test -n 131615c354d0 2023-10-11T11:18:23.974 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test -z 131615c354d0 -o 131615c354d0 = 131615c354d0 2023-10-11T11:18:23.975 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ current_state=present 2023-10-11T11:18:23.975 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test present = present 2023-10-11T11:18:23.975 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:18:23.975 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ disable_mirror cluster2 mirror image_10 2023-10-11T11:18:23.975 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:18:23.976 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:18:23.976 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_10 2023-10-11T11:18:23.976 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_10 2023-10-11T11:18:25.086 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~173s 2023-10-11T11:18:29.703 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi111.stderr:2023-10-11T11:18:29.702+0000 7f1e68fcb700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x556984624ea0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-10-11T11:18:29.704 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi111.stderr:2023-10-11T11:18:29.702+0000 7f1e6efd7700 -1 rbd::mirror::ImageReplayer: 0x5569846da780 [3/8bca97e8-61d3-4c2b-aea1-515700fbe799] operator(): start failed: (85) Interrupted system call should be restarted 2023-10-11T11:18:30.489 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~178s 2023-10-11T11:18:35.892 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~184s 2023-10-11T11:18:41.295 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~189s 2023-10-11T11:18:46.699 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~195s 2023-10-11T11:18:52.103 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~200s 2023-10-11T11:18:54.729 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi111.stderr:2023-10-11T11:18:54.728+0000 7f2c296d2700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55c1e972f400 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=386 2023-10-11T11:18:54.729 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi111.stderr:2023-10-11T11:18:54.728+0000 7f2c296d2700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55c1e972f400 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=390, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=9a182971-2e50-451f-a839-7c08d3b6157c, primary_snap_id=182, last_copied_object_number=128, snap_seqs={386=18446744073709551614}] 2023-10-11T11:18:54.733 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:Mirroring disabled 2023-10-11T11:18:54.747 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_present cluster1 mirror image_10 deleted 2023-10-11T11:18:54.747 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:18:54.748 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:18:54.748 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_10 2023-10-11T11:18:54.748 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local state=deleted 2023-10-11T11:18:54.748 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image_id= 2023-10-11T11:18:54.748 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local s 2023-10-11T11:18:54.749 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test -n '' 2023-10-11T11:18:54.749 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ get_image_id cluster1 mirror image_10 2023-10-11T11:18:54.749 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:18:54.749 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local pool=mirror 2023-10-11T11:18:54.749 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local image=image_10 2023-10-11T11:18:54.750 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ rbd --cluster=cluster1 info mirror/image_10 2023-10-11T11:18:54.750 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-10-11T11:18:54.783 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd: error opening image image_10: (2) No such file or directory 2023-10-11T11:18:54.787 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ image_id= 2023-10-11T11:18:54.787 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-10-11T11:18:54.787 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0.1 2023-10-11T11:18:54.889 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test_image_present cluster1 mirror image_10 deleted '' 2023-10-11T11:18:54.889 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:18:54.889 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:18:54.889 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_10 2023-10-11T11:18:54.890 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local test_state=deleted 2023-10-11T11:18:54.890 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image_id= 2023-10-11T11:18:54.890 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_state=deleted 2023-10-11T11:18:54.890 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local current_image_id 2023-10-11T11:18:54.890 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ get_image_id cluster1 mirror image_10 2023-10-11T11:18:54.891 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:18:54.891 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local pool=mirror 2023-10-11T11:18:54.891 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local image=image_10 2023-10-11T11:18:54.891 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ rbd --cluster=cluster1 info mirror/image_10 2023-10-11T11:18:54.891 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-10-11T11:18:54.923 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:rbd: error opening image image_10: (2) No such file or directory 2023-10-11T11:18:54.927 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ current_image_id= 2023-10-11T11:18:54.928 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test -n '' 2023-10-11T11:18:54.928 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test deleted = deleted 2023-10-11T11:18:54.928 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:18:54.928 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ purge_snapshots cluster2 mirror image_10 2023-10-11T11:18:54.928 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:18:54.929 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:18:54.929 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_10 2023-10-11T11:18:54.929 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster cluster2 snap purge mirror/image_10 2023-10-11T11:18:54.971 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_image_retry cluster2 mirror image_10 2023-10-11T11:18:54.971 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:18:54.971 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:18:54.971 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_10 2023-10-11T11:18:54.971 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0 1 2 4 8 16 32 2023-10-11T11:18:54.972 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0 2023-10-11T11:18:54.973 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ remove_image cluster2 mirror image_10 2023-10-11T11:18:54.973 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:18:54.973 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:18:54.973 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local image=image_10 2023-10-11T11:18:54.973 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_10 2023-10-11T11:18:55.013 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rbd --cluster=cluster2 rm mirror/image_10 2023-10-11T11:18:55.253 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2023-10-11T11:18:55.256 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:18:55.256 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ testlog 'TEST: check if removed images'\'' OMAP are removed' 2023-10-11T11:18:55.258 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ tee -a /tmp/tmp.nGgfBEp0YO/rbd-mirror.test.log 2023-10-11T11:18:55.258 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ date '+%F %T' 2023-10-11T11:18:55.259 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ echo 2023-10-11 11:18:55 TEST: check if removed 'images'\''' OMAP are removed 2023-10-11T11:18:55.259 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:2023-10-11 11:18:55 TEST: check if removed images' OMAP are removed 2023-10-11T11:18:55.259 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_in_omap cluster1 mirror 2023-10-11T11:18:55.259 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:18:55.259 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:18:55.260 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_omap_keys cluster1 mirror rbd_mirroring status_global 2023-10-11T11:18:55.260 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:18:55.260 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:18:55.260 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local obj_name=rbd_mirroring 2023-10-11T11:18:55.260 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local filter=status_global 2023-10-11T11:18:55.261 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2023-10-11T11:18:55.261 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0 2023-10-11T11:18:55.261 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ set +e 2023-10-11T11:18:55.262 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ count_omap_keys_with_filter cluster1 mirror rbd_mirroring status_global 2023-10-11T11:18:55.263 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:18:55.263 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local pool=mirror 2023-10-11T11:18:55.263 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local obj_name=rbd_mirroring 2023-10-11T11:18:55.263 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local filter=status_global 2023-10-11T11:18:55.263 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ list_omap_keys cluster1 mirror rbd_mirroring 2023-10-11T11:18:55.264 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:18:55.264 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local pool=mirror 2023-10-11T11:18:55.264 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local obj_name=rbd_mirroring 2023-10-11T11:18:55.264 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ grep -c status_global 2023-10-11T11:18:55.264 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ rados --cluster cluster1 -p mirror listomapkeys rbd_mirroring 2023-10-11T11:18:55.292 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 0 = 0 2023-10-11T11:18:55.293 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ error_code=0 2023-10-11T11:18:55.293 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ set -e 2023-10-11T11:18:55.293 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' 0 -eq 0 ']' 2023-10-11T11:18:55.293 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:18:55.294 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_omap_keys cluster1 mirror rbd_mirroring image_ 2023-10-11T11:18:55.294 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:18:55.294 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:18:55.294 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local obj_name=rbd_mirroring 2023-10-11T11:18:55.294 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local filter=image_ 2023-10-11T11:18:55.295 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2023-10-11T11:18:55.295 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0 2023-10-11T11:18:55.295 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ set +e 2023-10-11T11:18:55.296 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ count_omap_keys_with_filter cluster1 mirror rbd_mirroring image_ 2023-10-11T11:18:55.297 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:18:55.297 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local pool=mirror 2023-10-11T11:18:55.297 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local obj_name=rbd_mirroring 2023-10-11T11:18:55.297 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local filter=image_ 2023-10-11T11:18:55.297 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ list_omap_keys cluster1 mirror rbd_mirroring 2023-10-11T11:18:55.298 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:18:55.298 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local pool=mirror 2023-10-11T11:18:55.298 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local obj_name=rbd_mirroring 2023-10-11T11:18:55.298 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ grep -c image_ 2023-10-11T11:18:55.299 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ rados --cluster cluster1 -p mirror listomapkeys rbd_mirroring 2023-10-11T11:18:55.327 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 0 = 0 2023-10-11T11:18:55.327 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ error_code=0 2023-10-11T11:18:55.327 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ set -e 2023-10-11T11:18:55.328 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' 0 -eq 0 ']' 2023-10-11T11:18:55.328 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:18:55.328 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_omap_keys cluster1 mirror rbd_mirror_leader image_map 2023-10-11T11:18:55.328 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:18:55.328 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:18:55.329 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local obj_name=rbd_mirror_leader 2023-10-11T11:18:55.329 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local filter=image_map 2023-10-11T11:18:55.329 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2023-10-11T11:18:55.329 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0 2023-10-11T11:18:55.329 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ set +e 2023-10-11T11:18:55.331 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ count_omap_keys_with_filter cluster1 mirror rbd_mirror_leader image_map 2023-10-11T11:18:55.331 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:18:55.331 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local pool=mirror 2023-10-11T11:18:55.331 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local obj_name=rbd_mirror_leader 2023-10-11T11:18:55.331 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local filter=image_map 2023-10-11T11:18:55.332 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ list_omap_keys cluster1 mirror rbd_mirror_leader 2023-10-11T11:18:55.332 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:18:55.332 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local pool=mirror 2023-10-11T11:18:55.332 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ grep -c image_map 2023-10-11T11:18:55.332 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local obj_name=rbd_mirror_leader 2023-10-11T11:18:55.333 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ rados --cluster cluster1 -p mirror listomapkeys rbd_mirror_leader 2023-10-11T11:18:55.361 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 1 = 0 2023-10-11T11:18:55.361 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ error_code=1 2023-10-11T11:18:55.361 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ set -e 2023-10-11T11:18:55.361 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' 1 -eq 0 ']' 2023-10-11T11:18:55.362 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2023-10-11T11:18:55.362 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 1 2023-10-11T11:18:56.361 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ set +e 2023-10-11T11:18:56.363 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ count_omap_keys_with_filter cluster1 mirror rbd_mirror_leader image_map 2023-10-11T11:18:56.363 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:18:56.363 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local pool=mirror 2023-10-11T11:18:56.363 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local obj_name=rbd_mirror_leader 2023-10-11T11:18:56.363 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local filter=image_map 2023-10-11T11:18:56.364 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ list_omap_keys cluster1 mirror rbd_mirror_leader 2023-10-11T11:18:56.364 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:18:56.364 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local pool=mirror 2023-10-11T11:18:56.364 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local obj_name=rbd_mirror_leader 2023-10-11T11:18:56.364 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ rados --cluster cluster1 -p mirror listomapkeys rbd_mirror_leader 2023-10-11T11:18:56.365 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ grep -c image_map 2023-10-11T11:18:56.401 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 1 = 0 2023-10-11T11:18:56.401 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ error_code=1 2023-10-11T11:18:56.401 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ set -e 2023-10-11T11:18:56.401 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' 1 -eq 0 ']' 2023-10-11T11:18:56.401 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2023-10-11T11:18:56.402 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 2 2023-10-11T11:18:57.507 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~205s 2023-10-11T11:18:58.402 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ set +e 2023-10-11T11:18:58.403 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ count_omap_keys_with_filter cluster1 mirror rbd_mirror_leader image_map 2023-10-11T11:18:58.403 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:18:58.403 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local pool=mirror 2023-10-11T11:18:58.404 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local obj_name=rbd_mirror_leader 2023-10-11T11:18:58.404 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local filter=image_map 2023-10-11T11:18:58.404 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ list_omap_keys cluster1 mirror rbd_mirror_leader 2023-10-11T11:18:58.404 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster1 2023-10-11T11:18:58.405 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local pool=mirror 2023-10-11T11:18:58.405 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local obj_name=rbd_mirror_leader 2023-10-11T11:18:58.405 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ rados --cluster cluster1 -p mirror listomapkeys rbd_mirror_leader 2023-10-11T11:18:58.405 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ grep -c image_map 2023-10-11T11:18:58.443 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 0 = 0 2023-10-11T11:18:58.443 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ error_code=0 2023-10-11T11:18:58.443 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ set -e 2023-10-11T11:18:58.444 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' 0 -eq 0 ']' 2023-10-11T11:18:58.444 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:18:58.444 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_image_in_omap cluster2 mirror 2023-10-11T11:18:58.444 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:18:58.444 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:18:58.445 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_omap_keys cluster2 mirror rbd_mirroring status_global 2023-10-11T11:18:58.445 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:18:58.445 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:18:58.445 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local obj_name=rbd_mirroring 2023-10-11T11:18:58.445 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local filter=status_global 2023-10-11T11:18:58.446 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2023-10-11T11:18:58.446 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0 2023-10-11T11:18:58.446 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ set +e 2023-10-11T11:18:58.446 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ count_omap_keys_with_filter cluster2 mirror rbd_mirroring status_global 2023-10-11T11:18:58.446 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster2 2023-10-11T11:18:58.446 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local pool=mirror 2023-10-11T11:18:58.447 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local obj_name=rbd_mirroring 2023-10-11T11:18:58.447 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local filter=status_global 2023-10-11T11:18:58.447 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ list_omap_keys cluster2 mirror rbd_mirroring 2023-10-11T11:18:58.447 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster2 2023-10-11T11:18:58.447 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local pool=mirror 2023-10-11T11:18:58.448 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local obj_name=rbd_mirroring 2023-10-11T11:18:58.448 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ rados --cluster cluster2 -p mirror listomapkeys rbd_mirroring 2023-10-11T11:18:58.448 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ grep -c status_global 2023-10-11T11:18:58.478 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 0 = 0 2023-10-11T11:18:58.478 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ error_code=0 2023-10-11T11:18:58.479 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ set -e 2023-10-11T11:18:58.479 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' 0 -eq 0 ']' 2023-10-11T11:18:58.479 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:18:58.479 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_omap_keys cluster2 mirror rbd_mirroring image_ 2023-10-11T11:18:58.479 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:18:58.480 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:18:58.480 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local obj_name=rbd_mirroring 2023-10-11T11:18:58.480 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local filter=image_ 2023-10-11T11:18:58.480 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2023-10-11T11:18:58.480 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0 2023-10-11T11:18:58.481 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ set +e 2023-10-11T11:18:58.482 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ count_omap_keys_with_filter cluster2 mirror rbd_mirroring image_ 2023-10-11T11:18:58.482 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster2 2023-10-11T11:18:58.482 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local pool=mirror 2023-10-11T11:18:58.482 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local obj_name=rbd_mirroring 2023-10-11T11:18:58.482 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local filter=image_ 2023-10-11T11:18:58.483 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ list_omap_keys cluster2 mirror rbd_mirroring 2023-10-11T11:18:58.483 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster2 2023-10-11T11:18:58.483 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local pool=mirror 2023-10-11T11:18:58.483 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local obj_name=rbd_mirroring 2023-10-11T11:18:58.483 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ rados --cluster cluster2 -p mirror listomapkeys rbd_mirroring 2023-10-11T11:18:58.483 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ grep -c image_ 2023-10-11T11:18:58.513 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 0 = 0 2023-10-11T11:18:58.513 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ error_code=0 2023-10-11T11:18:58.514 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ set -e 2023-10-11T11:18:58.514 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' 0 -eq 0 ']' 2023-10-11T11:18:58.514 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:18:58.514 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ wait_for_omap_keys cluster2 mirror rbd_mirror_leader image_map 2023-10-11T11:18:58.515 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:18:58.515 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local pool=mirror 2023-10-11T11:18:58.515 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local obj_name=rbd_mirror_leader 2023-10-11T11:18:58.515 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local filter=image_map 2023-10-11T11:18:58.515 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2023-10-11T11:18:58.515 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ sleep 0 2023-10-11T11:18:58.517 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ set +e 2023-10-11T11:18:58.517 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ count_omap_keys_with_filter cluster2 mirror rbd_mirror_leader image_map 2023-10-11T11:18:58.517 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster2 2023-10-11T11:18:58.517 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local pool=mirror 2023-10-11T11:18:58.518 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local obj_name=rbd_mirror_leader 2023-10-11T11:18:58.518 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local filter=image_map 2023-10-11T11:18:58.518 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ list_omap_keys cluster2 mirror rbd_mirror_leader 2023-10-11T11:18:58.518 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local cluster=cluster2 2023-10-11T11:18:58.519 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local pool=mirror 2023-10-11T11:18:58.519 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ grep -c image_map 2023-10-11T11:18:58.519 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ local obj_name=rbd_mirror_leader 2023-10-11T11:18:58.519 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ rados --cluster cluster2 -p mirror listomapkeys rbd_mirror_leader 2023-10-11T11:18:58.547 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test 0 = 0 2023-10-11T11:18:58.548 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ error_code=0 2023-10-11T11:18:58.548 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ set -e 2023-10-11T11:18:58.548 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' 0 -eq 0 ']' 2023-10-11T11:18:58.548 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 0 2023-10-11T11:18:58.548 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ cleanup 0 2023-10-11T11:18:58.549 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local error_code=0 2023-10-11T11:18:58.549 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ set +e 2023-10-11T11:18:58.549 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' 0 -ne 0 ']' 2023-10-11T11:18:58.549 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -z '' ']' 2023-10-11T11:18:58.549 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster instance 2023-10-11T11:18:58.550 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ CEPH_ARGS= 2023-10-11T11:18:58.550 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ ceph --cluster cluster1 osd pool rm mirror mirror --yes-i-really-really-mean-it 2023-10-11T11:18:59.893 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi111.stderr:2023-10-11T11:18:59.892+0000 7f1e627be700 -1 rbd::mirror::RemotePoolPoller: 0x556982e7bad0 handle_mirror_peer_ping: failed to ping remote mirror peer: (2) No such file or directory 2023-10-11T11:18:59.894 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi111.stderr:2023-10-11T11:18:59.892+0000 7f1e627be700 -1 rbd::mirror::RemotePoolPoller: 0x556982e7bad0 handle_mirror_peer_list: failed to retrieve mirror peers: (2) No such file or directory 2023-10-11T11:18:59.894 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi111.stderr:2023-10-11T11:18:59.892+0000 7f1e627be700 -1 rbd::mirror::PoolReplayer: 0x556983c74f00 handle_remote_pool_meta_updated: remote pool metadata updated unexpectedly 2023-10-11T11:18:59.896 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi111.stderr:2023-10-11T11:18:59.895+0000 7f1e627be700 -1 rbd::mirror::PoolWatcher: 0x556982e772c0 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2023-10-11T11:18:59.896 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi111.stderr:2023-10-11T11:18:59.896+0000 7f1e61fbd700 -1 rbd::mirror::PoolWatcher: 0x556982e76000 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2023-10-11T11:18:59.908 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi111.stderr:2023-10-11T11:18:59.907+0000 7f1e627be700 -1 rbd::mirror::MirrorStatusUpdater 0x556983f965a0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-10-11T11:18:59.908 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi111.stderr:2023-10-11T11:18:59.907+0000 7f1e6f7d8700 -1 rbd::mirror::NamespaceReplayer: 0x556983f489c0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-10-11T11:18:59.913 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi111.stderr:2023-10-11T11:18:59.912+0000 7f1e61fbd700 -1 rbd::mirror::MirrorStatusUpdater 0x556983f96360 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-10-11T11:18:59.913 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi111.stderr:2023-10-11T11:18:59.912+0000 7f1e6f7d8700 -1 rbd::mirror::NamespaceReplayer: 0x556983f48820 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-10-11T11:18:59.981 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:pool 'mirror' does not exist 2023-10-11T11:18:59.992 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ CEPH_ARGS= 2023-10-11T11:18:59.993 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ ceph --cluster cluster2 osd pool rm mirror mirror --yes-i-really-really-mean-it 2023-10-11T11:19:00.794 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi111.stderr:2023-10-11T11:19:00.792+0000 7f1e68fcb700 -1 rbd::mirror::image_deleter::TrashWatcher: 0x556984122000 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2023-10-11T11:19:00.794 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi111.stderr:2023-10-11T11:19:00.793+0000 7f1e68fcb700 -1 rbd::mirror::PoolWatcher: 0x556982e770e0 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2023-10-11T11:19:00.794 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi111.stderr:2023-10-11T11:19:00.793+0000 7f1e627be700 -1 rbd::mirror::PoolWatcher: 0x556982e774a0 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2023-10-11T11:19:00.795 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi111.stderr:2023-10-11T11:19:00.793+0000 7f1e6f7d8700 -1 rbd::mirror::LeaderWatcher: 0x556983f22f00 handle_notify_lock_released: error notifying leader lock released: (2) No such file or directory 2023-10-11T11:19:00.795 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi111.stderr:2023-10-11T11:19:00.793+0000 7f1e6f7d8700 -1 rbd::mirror::LeaderWatcher: 0x556983f22f00 handle_shut_down_leader_lock: error shutting down leader lock: (2) No such file or directory 2023-10-11T11:19:00.795 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi111.stderr:2023-10-11T11:19:00.793+0000 7f1e6efd7700 -1 rbd::mirror::LeaderWatcher: 0x556983f22f00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2023-10-11T11:19:00.796 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi111.stderr:2023-10-11T11:19:00.794+0000 7f1e6efd7700 -1 rbd::mirror::InstanceWatcher: 0x556983d92e00 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5059 object: (2) No such file or directory 2023-10-11T11:19:00.796 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi111.stderr:2023-10-11T11:19:00.794+0000 7f1e68fcb700 -1 rbd::mirror::InstanceWatcher: 0x556983d92e00 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-10-11T11:19:00.796 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi111.stderr:2023-10-11T11:19:00.794+0000 7f1e627be700 -1 rbd::mirror::MirrorStatusUpdater 0x556983d0a6c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-10-11T11:19:00.797 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi111.stderr:2023-10-11T11:19:00.794+0000 7f1e6efd7700 -1 rbd::mirror::NamespaceReplayer: 0x556983f48000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-10-11T11:19:00.797 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi111.stderr:2023-10-11T11:19:00.795+0000 7f1e687ca700 -1 rbd::mirror::MirrorStatusUpdater 0x5569848ec6c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-10-11T11:19:00.797 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi111.stderr:2023-10-11T11:19:00.795+0000 7f1e6f7d8700 -1 rbd::mirror::NamespaceReplayer: 0x556983f48000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-10-11T11:19:00.813 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:pool 'mirror' does not exist 2023-10-11T11:19:00.827 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ CEPH_ARGS= 2023-10-11T11:19:00.827 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ ceph --cluster cluster1 osd pool rm mirror_parent mirror_parent --yes-i-really-really-mean-it 2023-10-11T11:19:00.944 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi111.stderr:2023-10-11T11:19:00.944+0000 7f2c366ec700 -1 rbd::mirror::LeaderWatcher: 0x55c1e6888f00 handle_notify_heartbeat: error notifying heartbeat: (2) No such file or directory, releasing leader 2023-10-11T11:19:00.946 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi111.stderr:2023-10-11T11:19:00.944+0000 7f2c2fedf700 -1 rbd::mirror::image_deleter::TrashWatcher: 0x55c1e688a6c0 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2023-10-11T11:19:00.946 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi111.stderr:2023-10-11T11:19:00.944+0000 7f2c306e0700 -1 rbd::mirror::image_deleter::TrashWatcher: 0x55c1e6cd2240 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2023-10-11T11:19:00.946 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi111.stderr:2023-10-11T11:19:00.944+0000 7f2c306e0700 -1 rbd::mirror::image_deleter::TrashWatcher: 0x55c1e688bb00 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2023-10-11T11:19:00.946 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi111.stderr:2023-10-11T11:19:00.944+0000 7f2c2fedf700 -1 rbd::mirror::PoolWatcher: 0x55c1e57e4000 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2023-10-11T11:19:00.946 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi111.stderr:2023-10-11T11:19:00.944+0000 7f2c2fedf700 -1 rbd::mirror::PoolWatcher: 0x55c1e57e45a0 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2023-10-11T11:19:00.947 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi111.stderr:2023-10-11T11:19:00.944+0000 7f2c2fedf700 -1 rbd::mirror::PoolWatcher: 0x55c1e57e43c0 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2023-10-11T11:19:00.947 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi111.stderr:2023-10-11T11:19:00.944+0000 7f2c296d2700 -1 rbd::mirror::PoolWatcher: 0x55c1e57e54a0 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2023-10-11T11:19:00.947 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi111.stderr:2023-10-11T11:19:00.944+0000 7f2c296d2700 -1 rbd::mirror::PoolWatcher: 0x55c1e57e52c0 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2023-10-11T11:19:00.947 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi111.stderr:2023-10-11T11:19:00.944+0000 7f2c296d2700 -1 rbd::mirror::PoolWatcher: 0x55c1e57e50e0 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2023-10-11T11:19:00.947 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi111.stderr:2023-10-11T11:19:00.945+0000 7f2c366ec700 -1 rbd::mirror::LeaderWatcher: 0x55c1e6888f00 handle_notify_lock_released: error notifying leader lock released: (2) No such file or directory 2023-10-11T11:19:00.948 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi111.stderr:2023-10-11T11:19:00.945+0000 7f2c366ec700 -1 rbd::mirror::LeaderWatcher: 0x55c1e6888f00 handle_release_leader_lock: error releasing lock: (2) No such file or directory 2023-10-11T11:19:01.833 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi111.stderr:2023-10-11T11:19:01.831+0000 7feb18891700 -1 rbd::mirror::RemotePoolPoller: 0x559e0ec75ad0 handle_mirror_peer_ping: failed to ping remote mirror peer: (2) No such file or directory 2023-10-11T11:19:01.833 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi111.stderr:2023-10-11T11:19:01.832+0000 7feb18891700 -1 rbd::mirror::RemotePoolPoller: 0x559e0ec75ad0 handle_mirror_peer_list: failed to retrieve mirror peers: (2) No such file or directory 2023-10-11T11:19:01.833 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi111.stderr:2023-10-11T11:19:01.832+0000 7feb18891700 -1 rbd::mirror::PoolReplayer: 0x559e0fa72f00 handle_remote_pool_meta_updated: remote pool metadata updated unexpectedly 2023-10-11T11:19:01.834 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi111.stderr:2023-10-11T11:19:01.832+0000 7feb260ac700 -1 rbd::mirror::InstanceWatcher: 0x559e0fd67500 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5834 object: (2) No such file or directory 2023-10-11T11:19:01.835 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi111.stderr:2023-10-11T11:19:01.832+0000 7feb258ab700 -1 rbd::mirror::InstanceWatcher: 0x559e10208700 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5834 object: (2) No such file or directory 2023-10-11T11:19:01.835 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi111.stderr:2023-10-11T11:19:01.833+0000 7feb1f89f700 -1 rbd::mirror::InstanceWatcher: 0x559e0fd67500 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-10-11T11:19:01.835 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi111.stderr:2023-10-11T11:19:01.833+0000 7feb1f09e700 -1 rbd::mirror::InstanceWatcher: 0x559e10208700 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-10-11T11:19:01.835 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi111.stderr:2023-10-11T11:19:01.833+0000 7feb19092700 -1 rbd::mirror::MirrorStatusUpdater 0x559e0fdac5a0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-10-11T11:19:01.835 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi111.stderr:2023-10-11T11:19:01.833+0000 7feb258ab700 -1 rbd::mirror::NamespaceReplayer: 0x559e0fd5c820 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-10-11T11:19:01.836 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi111.stderr:2023-10-11T11:19:01.833+0000 7feb18891700 -1 rbd::mirror::MirrorStatusUpdater 0x559e0fdac360 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-10-11T11:19:01.836 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi111.stderr:2023-10-11T11:19:01.833+0000 7feb258ab700 -1 rbd::mirror::NamespaceReplayer: 0x559e0fd5c9c0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-10-11T11:19:01.836 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi111.stderr:2023-10-11T11:19:01.833+0000 7feb1f89f700 -1 rbd::mirror::MirrorStatusUpdater 0x559e0fdac000 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-10-11T11:19:01.836 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi111.stderr:2023-10-11T11:19:01.833+0000 7feb1f89f700 -1 rbd::mirror::MirrorStatusUpdater 0x559e0fdac120 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-10-11T11:19:01.836 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi111.stderr:2023-10-11T11:19:01.833+0000 7feb260ac700 -1 rbd::mirror::NamespaceReplayer: 0x559e0fd5c820 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-10-11T11:19:01.836 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi111.stderr:2023-10-11T11:19:01.833+0000 7feb260ac700 -1 rbd::mirror::NamespaceReplayer: 0x559e0fd5c9c0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-10-11T11:19:01.861 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi111.stderr:2023-10-11T11:19:01.860+0000 7feb0b877700 -1 rbd::mirror::RemotePoolPoller: 0x559e0ec75ee0 handle_mirror_peer_ping: failed to ping remote mirror peer: (2) No such file or directory 2023-10-11T11:19:01.861 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi111.stderr:2023-10-11T11:19:01.860+0000 7feb0b076700 -1 rbd::mirror::RemotePoolPoller: 0x559e0ec75ee0 handle_mirror_peer_list: failed to retrieve mirror peers: (2) No such file or directory 2023-10-11T11:19:01.861 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi111.stderr:2023-10-11T11:19:01.860+0000 7feb0b076700 -1 rbd::mirror::PoolReplayer: 0x559e0fd31200 handle_remote_pool_meta_updated: remote pool metadata updated unexpectedly 2023-10-11T11:19:01.968 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi111.stderr:2023-10-11T11:19:01.968+0000 7feb258ab700 -1 rbd::mirror::LeaderWatcher: 0x559e0fd30f00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2023-10-11T11:19:01.976 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi111.stderr:2023-10-11T11:19:01.975+0000 7feb258ab700 -1 rbd::mirror::InstanceWatcher: 0x559e0fba2e00 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5834 object: (2) No such file or directory 2023-10-11T11:19:01.976 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi111.stderr:2023-10-11T11:19:01.975+0000 7feb1f89f700 -1 rbd::mirror::InstanceWatcher: 0x559e0fba2e00 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-10-11T11:19:01.977 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi111.stderr:2023-10-11T11:19:01.976+0000 7feb19092700 -1 rbd::mirror::MirrorStatusUpdater 0x559e0fb2c5a0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-10-11T11:19:01.978 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi111.stderr:2023-10-11T11:19:01.976+0000 7feb258ab700 -1 rbd::mirror::NamespaceReplayer: 0x559e0fd5c000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-10-11T11:19:01.978 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi111.stderr:2023-10-11T11:19:01.977+0000 7feb1f09e700 -1 rbd::mirror::MirrorStatusUpdater 0x559e1072c7e0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-10-11T11:19:01.979 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi111.stderr:2023-10-11T11:19:01.977+0000 7feb260ac700 -1 rbd::mirror::NamespaceReplayer: 0x559e0fd5c000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-10-11T11:19:01.986 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi111.stderr:2023-10-11T11:19:01.985+0000 7feb324c08c0 -1 rbd::mirror::Mirror: 0x559e0ef11c20 update_pool_replayers: restarting failed pool replayer for uuid: 216c798c-be92-4e54-a14c-4f00e5085964 cluster: cluster1 client: client.mirror 2023-10-11T11:19:02.037 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:pool 'mirror_parent' does not exist 2023-10-11T11:19:02.049 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ CEPH_ARGS= 2023-10-11T11:19:02.050 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ ceph --cluster cluster2 osd pool rm mirror_parent mirror_parent --yes-i-really-really-mean-it 2023-10-11T11:19:02.073 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi111.stderr:2023-10-11T11:19:02.072+0000 7feb0b877700 -1 rbd::mirror::PoolWatcher: 0x559e0ec703c0 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2023-10-11T11:19:02.346 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi111.stderr:2023-10-11T11:19:02.345+0000 7feb0b877700 -1 rbd::mirror::MirrorStatusUpdater 0x559e0fdac6c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-10-11T11:19:02.346 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi111.stderr:2023-10-11T11:19:02.345+0000 7feb258ab700 -1 rbd::mirror::NamespaceReplayer: 0x559e101404e0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-10-11T11:19:02.386 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi111.stderr:2023-10-11T11:19:02.385+0000 7feb324c08c0 -1 rbd::mirror::PoolReplayer: 0x559e0fd31200 init: error accessing remote pool mirror_parent: (2) No such file or directory 2023-10-11T11:19:02.843 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi111.stderr:2023-10-11T11:19:02.842+0000 7f07f3ec9700 -1 rbd::mirror::LeaderWatcher: 0x558745969500 handle_notify_lock_released: error notifying leader lock released: (2) No such file or directory 2023-10-11T11:19:02.904 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:pool 'mirror_parent' does not exist 2023-10-11T11:19:02.909 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~211s 2023-10-11T11:19:02.914 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for cluster in "${CLUSTER1}" "${CLUSTER2}" 2023-10-11T11:19:02.914 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ stop_mirrors cluster1 2023-10-11T11:19:02.915 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1 2023-10-11T11:19:02.915 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local sig= 2023-10-11T11:19:02.915 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ seq 0 3 2023-10-11T11:19:02.919 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-10-11T11:19:02.921 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ stop_mirror cluster1:0 '' 2023-10-11T11:19:02.921 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1:0 2023-10-11T11:19:02.921 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local sig= 2023-10-11T11:19:02.922 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test -n 1 2023-10-11T11:19:02.922 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 2023-10-11T11:19:02.922 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-10-11T11:19:02.922 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ stop_mirror cluster1:1 '' 2023-10-11T11:19:02.923 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1:1 2023-10-11T11:19:02.923 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local sig= 2023-10-11T11:19:02.923 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test -n 1 2023-10-11T11:19:02.923 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 2023-10-11T11:19:02.923 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-10-11T11:19:02.923 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ stop_mirror cluster1:2 '' 2023-10-11T11:19:02.924 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1:2 2023-10-11T11:19:02.924 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local sig= 2023-10-11T11:19:02.924 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test -n 1 2023-10-11T11:19:02.924 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 2023-10-11T11:19:02.924 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-10-11T11:19:02.925 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ stop_mirror cluster1:3 '' 2023-10-11T11:19:02.925 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster1:3 2023-10-11T11:19:02.925 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local sig= 2023-10-11T11:19:02.925 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test -n 1 2023-10-11T11:19:02.925 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 2023-10-11T11:19:02.926 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for cluster in "${CLUSTER1}" "${CLUSTER2}" 2023-10-11T11:19:02.926 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ stop_mirrors cluster2 2023-10-11T11:19:02.926 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2 2023-10-11T11:19:02.926 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local sig= 2023-10-11T11:19:02.926 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:++ seq 0 3 2023-10-11T11:19:02.927 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-10-11T11:19:02.927 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ stop_mirror cluster2:0 '' 2023-10-11T11:19:02.927 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2:0 2023-10-11T11:19:02.927 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local sig= 2023-10-11T11:19:02.927 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test -n 1 2023-10-11T11:19:02.928 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 2023-10-11T11:19:02.928 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-10-11T11:19:02.928 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ stop_mirror cluster2:1 '' 2023-10-11T11:19:02.928 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2:1 2023-10-11T11:19:02.928 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local sig= 2023-10-11T11:19:02.928 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test -n 1 2023-10-11T11:19:02.929 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 2023-10-11T11:19:02.929 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-10-11T11:19:02.929 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ stop_mirror cluster2:2 '' 2023-10-11T11:19:02.929 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2:2 2023-10-11T11:19:02.929 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local sig= 2023-10-11T11:19:02.930 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test -n 1 2023-10-11T11:19:02.930 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 2023-10-11T11:19:02.930 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-10-11T11:19:02.930 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ stop_mirror cluster2:3 '' 2023-10-11T11:19:02.930 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local cluster=cluster2:3 2023-10-11T11:19:02.931 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ local sig= 2023-10-11T11:19:02.931 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test -n 1 2023-10-11T11:19:02.931 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ return 2023-10-11T11:19:02.931 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' -z 1 ']' 2023-10-11T11:19:02.931 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ test '' = /tmp/tmp.nGgfBEp0YO 2023-10-11T11:19:02.932 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ rm -Rf /tmp/tmp.nGgfBEp0YO 2023-10-11T11:19:02.932 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ '[' 0 -eq 0 ']' 2023-10-11T11:19:02.932 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stdout:OK 2023-10-11T11:19:02.934 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ echo OK 2023-10-11T11:19:02.934 INFO:tasks.workunit.cluster1.client.mirror.smithi111.stderr:+ exit 0 2023-10-11T11:19:02.934 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-10-11T11:19:02.935 DEBUG:teuthology.orchestra.run.smithi111:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp 2023-10-11T11:19:03.010 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi111.stderr:2023-10-11T11:19:03.009+0000 7f0d10e70700 -1 rbd::mirror::LeaderWatcher: 0x55e8ef6dd500 handle_notify_heartbeat: error notifying heartbeat: (2) No such file or directory, releasing leader 2023-10-11T11:19:03.010 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi111.stderr:2023-10-11T11:19:03.009+0000 7f0cfce48700 -1 rbd::mirror::image_deleter::TrashWatcher: 0x55e8efa20900 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2023-10-11T11:19:03.010 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi111.stderr:2023-10-11T11:19:03.009+0000 7f0cfce48700 -1 rbd::mirror::PoolWatcher: 0x55e8ee6925a0 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2023-10-11T11:19:03.011 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi111.stderr:2023-10-11T11:19:03.009+0000 7f0cf663b700 -1 rbd::mirror::PoolWatcher: 0x55e8ee6923c0 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2023-10-11T11:19:03.011 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi111.stderr:2023-10-11T11:19:03.010+0000 7f0d10e70700 -1 rbd::mirror::LeaderWatcher: 0x55e8ef6dd500 handle_notify_lock_released: error notifying leader lock released: (2) No such file or directory 2023-10-11T11:19:03.011 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi111.stderr:2023-10-11T11:19:03.010+0000 7f0d10e70700 -1 rbd::mirror::LeaderWatcher: 0x55e8ef6dd500 handle_release_leader_lock: error releasing lock: (2) No such file or directory 2023-10-11T11:19:03.014 INFO:tasks.workunit:Stopping ['rbd/rbd_mirror_stress.sh'] on cluster1.client.mirror... 2023-10-11T11:19:03.015 DEBUG:teuthology.orchestra.run.smithi111:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.cluster1.client.mirror /home/ubuntu/cephtest/clone.cluster1.client.mirror 2023-10-11T11:19:03.320 DEBUG:teuthology.parallel:result is None 2023-10-11T11:19:03.321 DEBUG:teuthology.orchestra.run.smithi111:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror 2023-10-11T11:19:03.348 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror 2023-10-11T11:19:03.348 DEBUG:teuthology.orchestra.run.smithi111:> rmdir -- /home/ubuntu/cephtest/mnt.cluster1.mirror 2023-10-11T11:19:03.415 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror 2023-10-11T11:19:03.416 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror-thrash 2023-10-11T11:19:03.427 INFO:tasks.rbd_mirror_thrash:joining rbd_mirror_thrash 2023-10-11T11:19:03.427 INFO:tasks.rbd_mirror_thrash:stat['kill'] = 13 2023-10-11T11:19:03.427 INFO:tasks.rbd_mirror_thrash:done joining 2023-10-11T11:19:03.428 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror-thrash 2023-10-11T11:19:03.447 INFO:tasks.rbd_mirror_thrash:joining rbd_mirror_thrash 2023-10-11T11:19:03.447 INFO:tasks.rbd_mirror_thrash:stat['kill'] = 16 2023-10-11T11:19:03.447 INFO:tasks.rbd_mirror_thrash:done joining 2023-10-11T11:19:03.448 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2023-10-11T11:19:03.459 DEBUG:tasks.rbd_mirror.cluster2.client.mirror.3:waiting for process to exit 2023-10-11T11:19:03.460 INFO:teuthology.orchestra.run:waiting for 300 2023-10-11T11:19:03.460 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi111.stderr:2023-10-11T11:19:03.459+0000 7f1e6ffd9700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 (PID: 118993) UID: 1000 2023-10-11T11:19:03.461 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi111.stderr:2023-10-11T11:19:03.460+0000 7f1e6efd7700 -1 rbd::mirror::LeaderWatcher: 0x556984226900 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2023-10-11T11:19:03.461 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi111.stderr:2023-10-11T11:19:03.460+0000 7f1e6efd7700 -1 rbd::mirror::InstanceWatcher: 0x556983e98380 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5065 object: (2) No such file or directory 2023-10-11T11:19:03.461 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi111.stderr:2023-10-11T11:19:03.460+0000 7f1e5b7b0700 -1 rbd::mirror::InstanceWatcher: 0x556983e98380 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-10-11T11:19:03.462 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi111.stderr:2023-10-11T11:19:03.461+0000 7f1e547a2700 -1 rbd::mirror::MirrorStatusUpdater 0x556983f966c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-10-11T11:19:03.462 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi111.stderr:2023-10-11T11:19:03.461+0000 7f1e6efd7700 -1 rbd::mirror::NamespaceReplayer: 0x5569843504e0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-10-11T11:19:03.462 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi111.stderr:2023-10-11T11:19:03.461+0000 7f1e5b7b0700 -1 rbd::mirror::MirrorStatusUpdater 0x556984190120 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-10-11T11:19:03.463 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi111.stderr:2023-10-11T11:19:03.461+0000 7f1e6f7d8700 -1 rbd::mirror::NamespaceReplayer: 0x5569843504e0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-10-11T11:19:03.473 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Stopped 2023-10-11T11:19:03.473 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2023-10-11T11:19:03.483 DEBUG:tasks.rbd_mirror.cluster2.client.mirror.2:waiting for process to exit 2023-10-11T11:19:03.484 INFO:teuthology.orchestra.run:waiting for 300 2023-10-11T11:19:03.514 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi111.stderr:2023-10-11T11:19:03.513+0000 7f58bebe3700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 (PID: 134913) UID: 1000 2023-10-11T11:19:03.515 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi111.stderr:2023-10-11T11:19:03.513+0000 7f58be3e2700 -1 rbd::mirror::LeaderWatcher: 0x559e7253b500 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2023-10-11T11:19:03.515 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi111.stderr:2023-10-11T11:19:03.514+0000 7f58be3e2700 -1 rbd::mirror::InstanceWatcher: 0x559e72229880 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6134 object: (2) No such file or directory 2023-10-11T11:19:03.515 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi111.stderr:2023-10-11T11:19:03.514+0000 7f58be3e2700 -1 rbd::mirror::InstanceWatcher: 0x559e723f3500 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6134 object: (2) No such file or directory 2023-10-11T11:19:03.515 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi111.stderr:2023-10-11T11:19:03.514+0000 7f58be3e2700 -1 rbd::mirror::InstanceWatcher: 0x559e72542e00 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6137 object: (2) No such file or directory 2023-10-11T11:19:03.516 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi111.stderr:2023-10-11T11:19:03.514+0000 7f58b73d4700 -1 rbd::mirror::InstanceWatcher: 0x559e72229880 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-10-11T11:19:03.516 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi111.stderr:2023-10-11T11:19:03.514+0000 7f58b73d4700 -1 rbd::mirror::InstanceWatcher: 0x559e723f3500 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-10-11T11:19:03.516 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi111.stderr:2023-10-11T11:19:03.514+0000 7f58aa3ba700 -1 rbd::mirror::InstanceWatcher: 0x559e72542e00 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-10-11T11:19:03.517 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi111.stderr:2023-10-11T11:19:03.514+0000 7f58b13c8700 -1 rbd::mirror::MirrorStatusUpdater 0x559e7243e7e0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-10-11T11:19:03.517 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi111.stderr:2023-10-11T11:19:03.514+0000 7f58b13c8700 -1 rbd::mirror::MirrorStatusUpdater 0x559e7243e360 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-10-11T11:19:03.518 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi111.stderr:2023-10-11T11:19:03.514+0000 7f58be3e2700 -1 rbd::mirror::NamespaceReplayer: 0x559e7243c000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-10-11T11:19:03.518 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi111.stderr:2023-10-11T11:19:03.514+0000 7f58be3e2700 -1 rbd::mirror::NamespaceReplayer: 0x559e7243c1a0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-10-11T11:19:03.518 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi111.stderr:2023-10-11T11:19:03.515+0000 7f58b73d4700 -1 rbd::mirror::MirrorStatusUpdater 0x559e7243e000 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-10-11T11:19:03.518 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi111.stderr:2023-10-11T11:19:03.515+0000 7f58b73d4700 -1 rbd::mirror::MirrorStatusUpdater 0x559e7243e120 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-10-11T11:19:03.518 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi111.stderr:2023-10-11T11:19:03.515+0000 7f58bdbe1700 -1 rbd::mirror::NamespaceReplayer: 0x559e7243c000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-10-11T11:19:03.518 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi111.stderr:2023-10-11T11:19:03.515+0000 7f58bdbe1700 -1 rbd::mirror::NamespaceReplayer: 0x559e7243c1a0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-10-11T11:19:03.519 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi111.stderr:2023-10-11T11:19:03.515+0000 7f58a3bad700 -1 rbd::mirror::MirrorStatusUpdater 0x559e7266e6c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-10-11T11:19:03.519 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi111.stderr:2023-10-11T11:19:03.515+0000 7f58be3e2700 -1 rbd::mirror::NamespaceReplayer: 0x559e7289c000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-10-11T11:19:03.519 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi111.stderr:2023-10-11T11:19:03.515+0000 7f58aa3ba700 -1 rbd::mirror::MirrorStatusUpdater 0x559e72662240 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-10-11T11:19:03.519 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi111.stderr:2023-10-11T11:19:03.515+0000 7f58bdbe1700 -1 rbd::mirror::NamespaceReplayer: 0x559e7289c000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-10-11T11:19:03.520 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi111.stderr:2023-10-11T11:19:03.517+0000 7f58be3e2700 -1 rbd::mirror::LeaderWatcher: 0x559e7219b500 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2023-10-11T11:19:03.520 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi111.stderr:2023-10-11T11:19:03.517+0000 7f58be3e2700 -1 rbd::mirror::InstanceWatcher: 0x559e72228e00 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6134 object: (2) No such file or directory 2023-10-11T11:19:03.521 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi111.stderr:2023-10-11T11:19:03.517+0000 7f58b73d4700 -1 rbd::mirror::InstanceWatcher: 0x559e72228e00 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-10-11T11:19:03.521 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi111.stderr:2023-10-11T11:19:03.518+0000 7f58b13c8700 -1 rbd::mirror::MirrorStatusUpdater 0x559e7235a480 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-10-11T11:19:03.521 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi111.stderr:2023-10-11T11:19:03.518+0000 7f58be3e2700 -1 rbd::mirror::NamespaceReplayer: 0x559e72336340 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-10-11T11:19:03.521 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi111.stderr:2023-10-11T11:19:03.518+0000 7f58b7bd5700 -1 rbd::mirror::MirrorStatusUpdater 0x559e72198900 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-10-11T11:19:03.521 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi111.stderr:2023-10-11T11:19:03.518+0000 7f58bdbe1700 -1 rbd::mirror::NamespaceReplayer: 0x559e72336340 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-10-11T11:19:03.529 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Stopped 2023-10-11T11:19:03.529 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2023-10-11T11:19:03.539 DEBUG:tasks.rbd_mirror.cluster2.client.mirror.1:waiting for process to exit 2023-10-11T11:19:03.539 INFO:teuthology.orchestra.run:waiting for 300 2023-10-11T11:19:03.570 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi111.stderr:2023-10-11T11:19:03.569+0000 7feb268ad700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 132600) UID: 1000 2023-10-11T11:19:03.580 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Stopped 2023-10-11T11:19:03.580 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2023-10-11T11:19:03.591 DEBUG:tasks.rbd_mirror.cluster2.client.mirror.0:waiting for process to exit 2023-10-11T11:19:03.591 INFO:teuthology.orchestra.run:waiting for 300 2023-10-11T11:19:03.621 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi111.stderr:2023-10-11T11:19:03.620+0000 7f07f46ca700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 140588) UID: 1000 2023-10-11T11:19:03.622 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi111.stderr:2023-10-11T11:19:03.621+0000 7f07f36c8700 -1 rbd::mirror::InstanceWatcher: 0x558745825500 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6387 object: (2) No such file or directory 2023-10-11T11:19:03.622 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi111.stderr:2023-10-11T11:19:03.621+0000 7f07f36c8700 -1 rbd::mirror::InstanceWatcher: 0x558745869c00 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6390 object: (2) No such file or directory 2023-10-11T11:19:03.622 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi111.stderr:2023-10-11T11:19:03.621+0000 7f07f3ec9700 -1 rbd::mirror::InstanceWatcher: 0x558745cf0700 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6387 object: (2) No such file or directory 2023-10-11T11:19:03.622 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi111.stderr:2023-10-11T11:19:03.621+0000 7f07ed6bc700 -1 rbd::mirror::InstanceWatcher: 0x558745825500 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-10-11T11:19:03.623 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi111.stderr:2023-10-11T11:19:03.621+0000 7f07df6a0700 -1 rbd::mirror::InstanceWatcher: 0x558745869c00 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-10-11T11:19:03.623 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi111.stderr:2023-10-11T11:19:03.621+0000 7f07ecebb700 -1 rbd::mirror::InstanceWatcher: 0x558745cf0700 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-10-11T11:19:03.624 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi111.stderr:2023-10-11T11:19:03.621+0000 7f07e6eaf700 -1 rbd::mirror::MirrorStatusUpdater 0x558745870360 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-10-11T11:19:03.624 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi111.stderr:2023-10-11T11:19:03.621+0000 7f07d9694700 -1 rbd::mirror::MirrorStatusUpdater 0x5587458706c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-10-11T11:19:03.624 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi111.stderr:2023-10-11T11:19:03.621+0000 7f07f3ec9700 -1 rbd::mirror::NamespaceReplayer: 0x5587458189c0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-10-11T11:19:03.625 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi111.stderr:2023-10-11T11:19:03.621+0000 7f07e66ae700 -1 rbd::mirror::MirrorStatusUpdater 0x5587458705a0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-10-11T11:19:03.625 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi111.stderr:2023-10-11T11:19:03.621+0000 7f07f3ec9700 -1 rbd::mirror::NamespaceReplayer: 0x558745c2c340 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-10-11T11:19:03.625 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi111.stderr:2023-10-11T11:19:03.621+0000 7f07f3ec9700 -1 rbd::mirror::NamespaceReplayer: 0x558745818b60 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-10-11T11:19:03.625 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi111.stderr:2023-10-11T11:19:03.621+0000 7f07ed6bc700 -1 rbd::mirror::MirrorStatusUpdater 0x558745870000 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-10-11T11:19:03.625 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi111.stderr:2023-10-11T11:19:03.621+0000 7f07ed6bc700 -1 rbd::mirror::MirrorStatusUpdater 0x558745870120 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-10-11T11:19:03.626 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi111.stderr:2023-10-11T11:19:03.621+0000 7f07f3ec9700 -1 rbd::mirror::NamespaceReplayer: 0x5587458189c0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-10-11T11:19:03.626 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi111.stderr:2023-10-11T11:19:03.621+0000 7f07f3ec9700 -1 rbd::mirror::NamespaceReplayer: 0x558745818b60 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-10-11T11:19:03.626 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi111.stderr:2023-10-11T11:19:03.623+0000 7f07f3ec9700 -1 rbd::mirror::LeaderWatcher: 0x5587455f9500 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2023-10-11T11:19:03.627 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi111.stderr:2023-10-11T11:19:03.623+0000 7f07f3ec9700 -1 rbd::mirror::InstanceWatcher: 0x55874566ae00 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6387 object: (2) No such file or directory 2023-10-11T11:19:03.627 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi111.stderr:2023-10-11T11:19:03.624+0000 7f07ecebb700 -1 rbd::mirror::InstanceWatcher: 0x55874566ae00 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-10-11T11:19:03.627 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi111.stderr:2023-10-11T11:19:03.624+0000 7f07e6eaf700 -1 rbd::mirror::MirrorStatusUpdater 0x5587455f65a0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-10-11T11:19:03.627 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi111.stderr:2023-10-11T11:19:03.624+0000 7f07f3ec9700 -1 rbd::mirror::NamespaceReplayer: 0x558745818000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-10-11T11:19:03.628 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi111.stderr:2023-10-11T11:19:03.624+0000 7f07ed6bc700 -1 rbd::mirror::MirrorStatusUpdater 0x5587461ee7e0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-10-11T11:19:03.628 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi111.stderr:2023-10-11T11:19:03.624+0000 7f07f36c8700 -1 rbd::mirror::NamespaceReplayer: 0x558745818000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-10-11T11:19:03.635 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Stopped 2023-10-11T11:19:03.636 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2023-10-11T11:19:03.648 DEBUG:tasks.rbd_mirror.cluster1.client.mirror.3:waiting for process to exit 2023-10-11T11:19:03.649 INFO:teuthology.orchestra.run:waiting for 300 2023-10-11T11:19:03.677 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi111.stderr:2023-10-11T11:19:03.676+0000 7f0d11671700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 (PID: 120083) UID: 1000 2023-10-11T11:19:03.677 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi111.stderr:2023-10-11T11:19:03.676+0000 7f0d10e70700 -1 rbd::mirror::LeaderWatcher: 0x55e8ef6dd500 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2023-10-11T11:19:03.679 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi111.stderr:2023-10-11T11:19:03.677+0000 7f0d10e70700 -1 rbd::mirror::InstanceWatcher: 0x55e8ef612700 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.4786 object: (2) No such file or directory 2023-10-11T11:19:03.679 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi111.stderr:2023-10-11T11:19:03.677+0000 7f0d10e70700 -1 rbd::mirror::InstanceWatcher: 0x55e8efa5a380 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.4786 object: (2) No such file or directory 2023-10-11T11:19:03.679 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi111.stderr:2023-10-11T11:19:03.677+0000 7f0d1066f700 -1 rbd::mirror::InstanceWatcher: 0x55e8efa4c380 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.4789 object: (2) No such file or directory 2023-10-11T11:19:03.680 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi111.stderr:2023-10-11T11:19:03.677+0000 7f0d0a663700 -1 rbd::mirror::InstanceWatcher: 0x55e8ef612700 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-10-11T11:19:03.680 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi111.stderr:2023-10-11T11:19:03.677+0000 7f0d09e62700 -1 rbd::mirror::InstanceWatcher: 0x55e8efa5a380 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-10-11T11:19:03.680 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi111.stderr:2023-10-11T11:19:03.677+0000 7f0cfce48700 -1 rbd::mirror::InstanceWatcher: 0x55e8efa4c380 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-10-11T11:19:03.680 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi111.stderr:2023-10-11T11:19:03.677+0000 7f0d03e56700 -1 rbd::mirror::MirrorStatusUpdater 0x55e8ef5fa7e0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-10-11T11:19:03.680 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi111.stderr:2023-10-11T11:19:03.677+0000 7f0d10e70700 -1 rbd::mirror::NamespaceReplayer: 0x55e8ef602000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-10-11T11:19:03.680 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi111.stderr:2023-10-11T11:19:03.677+0000 7f0d03e56700 -1 rbd::mirror::MirrorStatusUpdater 0x55e8ef5fa5a0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-10-11T11:19:03.681 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi111.stderr:2023-10-11T11:19:03.677+0000 7f0d10e70700 -1 rbd::mirror::NamespaceReplayer: 0x55e8ef6021a0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-10-11T11:19:03.681 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi111.stderr:2023-10-11T11:19:03.677+0000 7f0cf5e3a700 -1 rbd::mirror::MirrorStatusUpdater 0x55e8ef5fa900 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-10-11T11:19:03.681 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi111.stderr:2023-10-11T11:19:03.677+0000 7f0d1066f700 -1 rbd::mirror::NamespaceReplayer: 0x55e8ef602ea0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-10-11T11:19:03.681 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi111.stderr:2023-10-11T11:19:03.677+0000 7f0d0a663700 -1 rbd::mirror::MirrorStatusUpdater 0x55e8ef5fa240 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-10-11T11:19:03.681 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi111.stderr:2023-10-11T11:19:03.677+0000 7f0d10e70700 -1 rbd::mirror::NamespaceReplayer: 0x55e8ef602000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-10-11T11:19:03.682 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi111.stderr:2023-10-11T11:19:03.678+0000 7f0d0a663700 -1 rbd::mirror::MirrorStatusUpdater 0x55e8ef5fa360 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-10-11T11:19:03.682 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi111.stderr:2023-10-11T11:19:03.678+0000 7f0d1066f700 -1 rbd::mirror::NamespaceReplayer: 0x55e8ef6021a0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-10-11T11:19:03.682 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi111.stderr:2023-10-11T11:19:03.678+0000 7f0cfce48700 -1 rbd::mirror::MirrorStatusUpdater 0x55e8ef816360 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-10-11T11:19:03.682 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi111.stderr:2023-10-11T11:19:03.678+0000 7f0d10e70700 -1 rbd::mirror::NamespaceReplayer: 0x55e8ef602ea0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-10-11T11:19:03.683 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi111.stderr:2023-10-11T11:19:03.680+0000 7f0d1066f700 -1 rbd::mirror::LeaderWatcher: 0x55e8f01e7500 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2023-10-11T11:19:03.683 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi111.stderr:2023-10-11T11:19:03.680+0000 7f0d1066f700 -1 rbd::mirror::InstanceWatcher: 0x55e8f0234e00 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.4786 object: (2) No such file or directory 2023-10-11T11:19:03.683 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi111.stderr:2023-10-11T11:19:03.680+0000 7f0d09e62700 -1 rbd::mirror::InstanceWatcher: 0x55e8f0234e00 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-10-11T11:19:03.684 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi111.stderr:2023-10-11T11:19:03.680+0000 7f0d03655700 -1 rbd::mirror::MirrorStatusUpdater 0x55e8f01d06c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-10-11T11:19:03.684 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi111.stderr:2023-10-11T11:19:03.680+0000 7f0d1066f700 -1 rbd::mirror::NamespaceReplayer: 0x55e8f0340340 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-10-11T11:19:03.684 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi111.stderr:2023-10-11T11:19:03.681+0000 7f0d09e62700 -1 rbd::mirror::MirrorStatusUpdater 0x55e8f01286c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-10-11T11:19:03.684 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi111.stderr:2023-10-11T11:19:03.681+0000 7f0d10e70700 -1 rbd::mirror::NamespaceReplayer: 0x55e8f0340340 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-10-11T11:19:03.692 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Stopped 2023-10-11T11:19:03.692 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2023-10-11T11:19:03.706 DEBUG:tasks.rbd_mirror.cluster1.client.mirror.2:waiting for process to exit 2023-10-11T11:19:03.706 INFO:teuthology.orchestra.run:waiting for 300 2023-10-11T11:19:03.733 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi111.stderr:2023-10-11T11:19:03.732+0000 7f3fc6a2b700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 (PID: 136076) UID: 1000 2023-10-11T11:19:03.733 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi111.stderr:2023-10-11T11:19:03.732+0000 7f3fc622a700 -1 rbd::mirror::LeaderWatcher: 0x5604ed61ef00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2023-10-11T11:19:03.735 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi111.stderr:2023-10-11T11:19:03.733+0000 7f3fc5a29700 -1 rbd::mirror::InstanceWatcher: 0x5604ed0df500 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5665 object: (2) No such file or directory 2023-10-11T11:19:03.735 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi111.stderr:2023-10-11T11:19:03.733+0000 7f3fc622a700 -1 rbd::mirror::InstanceWatcher: 0x5604ed316380 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5665 object: (2) No such file or directory 2023-10-11T11:19:03.735 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi111.stderr:2023-10-11T11:19:03.733+0000 7f3fc622a700 -1 rbd::mirror::InstanceWatcher: 0x5604ed756000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5668 object: (2) No such file or directory 2023-10-11T11:19:03.736 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi111.stderr:2023-10-11T11:19:03.733+0000 7f3fbf21c700 -1 rbd::mirror::InstanceWatcher: 0x5604ed0df500 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-10-11T11:19:03.736 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi111.stderr:2023-10-11T11:19:03.733+0000 7f3fbfa1d700 -1 rbd::mirror::InstanceWatcher: 0x5604ed316380 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-10-11T11:19:03.736 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi111.stderr:2023-10-11T11:19:03.733+0000 7f3fb2202700 -1 rbd::mirror::InstanceWatcher: 0x5604ed756000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-10-11T11:19:03.736 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi111.stderr:2023-10-11T11:19:03.733+0000 7f3fb9210700 -1 rbd::mirror::MirrorStatusUpdater 0x5604ed2ec360 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-10-11T11:19:03.736 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi111.stderr:2023-10-11T11:19:03.733+0000 7f3fb9210700 -1 rbd::mirror::MirrorStatusUpdater 0x5604ed2ec5a0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-10-11T11:19:03.737 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi111.stderr:2023-10-11T11:19:03.733+0000 7f3fc622a700 -1 rbd::mirror::NamespaceReplayer: 0x5604ed28e820 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-10-11T11:19:03.737 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi111.stderr:2023-10-11T11:19:03.733+0000 7f3fc622a700 -1 rbd::mirror::NamespaceReplayer: 0x5604ed28e9c0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-10-11T11:19:03.737 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi111.stderr:2023-10-11T11:19:03.734+0000 7f3fbfa1d700 -1 rbd::mirror::MirrorStatusUpdater 0x5604ed2ec000 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-10-11T11:19:03.737 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi111.stderr:2023-10-11T11:19:03.734+0000 7f3fbfa1d700 -1 rbd::mirror::MirrorStatusUpdater 0x5604ed2ec120 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-10-11T11:19:03.737 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi111.stderr:2023-10-11T11:19:03.734+0000 7f3fc5a29700 -1 rbd::mirror::NamespaceReplayer: 0x5604ed28e820 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-10-11T11:19:03.737 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi111.stderr:2023-10-11T11:19:03.734+0000 7f3fc5a29700 -1 rbd::mirror::NamespaceReplayer: 0x5604ed28e9c0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-10-11T11:19:03.738 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi111.stderr:2023-10-11T11:19:03.734+0000 7f3fab9f5700 -1 rbd::mirror::MirrorStatusUpdater 0x5604ed754000 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-10-11T11:19:03.738 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi111.stderr:2023-10-11T11:19:03.734+0000 7f3fc622a700 -1 rbd::mirror::NamespaceReplayer: 0x5604ed602680 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-10-11T11:19:03.738 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi111.stderr:2023-10-11T11:19:03.734+0000 7f3fb2202700 -1 rbd::mirror::MirrorStatusUpdater 0x5604ed4e27e0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-10-11T11:19:03.738 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi111.stderr:2023-10-11T11:19:03.734+0000 7f3fc5a29700 -1 rbd::mirror::NamespaceReplayer: 0x5604ed602680 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-10-11T11:19:03.739 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi111.stderr:2023-10-11T11:19:03.736+0000 7f3fc622a700 -1 rbd::mirror::LeaderWatcher: 0x5604ed276900 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2023-10-11T11:19:03.739 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi111.stderr:2023-10-11T11:19:03.736+0000 7f3fc622a700 -1 rbd::mirror::InstanceWatcher: 0x5604ed0dee00 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5665 object: (2) No such file or directory 2023-10-11T11:19:03.740 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi111.stderr:2023-10-11T11:19:03.736+0000 7f3fbfa1d700 -1 rbd::mirror::InstanceWatcher: 0x5604ed0dee00 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-10-11T11:19:03.740 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi111.stderr:2023-10-11T11:19:03.737+0000 7f3fb8a0f700 -1 rbd::mirror::MirrorStatusUpdater 0x5604edc606c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-10-11T11:19:03.740 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi111.stderr:2023-10-11T11:19:03.737+0000 7f3fc622a700 -1 rbd::mirror::NamespaceReplayer: 0x5604ed28e000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-10-11T11:19:03.740 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi111.stderr:2023-10-11T11:19:03.737+0000 7f3fbfa1d700 -1 rbd::mirror::MirrorStatusUpdater 0x5604edc626c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-10-11T11:19:03.740 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi111.stderr:2023-10-11T11:19:03.737+0000 7f3fc5a29700 -1 rbd::mirror::NamespaceReplayer: 0x5604ed28e000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-10-11T11:19:03.748 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Stopped 2023-10-11T11:19:03.748 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2023-10-11T11:19:03.761 DEBUG:tasks.rbd_mirror.cluster1.client.mirror.1:waiting for process to exit 2023-10-11T11:19:03.761 INFO:teuthology.orchestra.run:waiting for 300 2023-10-11T11:19:03.789 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi111.stderr:2023-10-11T11:19:03.788+0000 7f2c376ee700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 121521) UID: 1000 2023-10-11T11:19:03.790 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi111.stderr:2023-10-11T11:19:03.789+0000 7f2c366ec700 -1 rbd::mirror::LeaderWatcher: 0x55c1e6c2cf00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2023-10-11T11:19:03.791 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi111.stderr:2023-10-11T11:19:03.789+0000 7f2c366ec700 -1 rbd::mirror::InstanceWatcher: 0x55c1e68c5500 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.4886 object: (2) No such file or directory 2023-10-11T11:19:03.792 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi111.stderr:2023-10-11T11:19:03.789+0000 7f2c366ec700 -1 rbd::mirror::InstanceWatcher: 0x55c1e6d82000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.4886 object: (2) No such file or directory 2023-10-11T11:19:03.792 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi111.stderr:2023-10-11T11:19:03.789+0000 7f2c366ec700 -1 rbd::mirror::InstanceWatcher: 0x55c1e690fc00 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.4889 object: (2) No such file or directory 2023-10-11T11:19:03.792 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi111.stderr:2023-10-11T11:19:03.789+0000 7f2c306e0700 -1 rbd::mirror::InstanceWatcher: 0x55c1e68c5500 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-10-11T11:19:03.792 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi111.stderr:2023-10-11T11:19:03.789+0000 7f2c306e0700 -1 rbd::mirror::InstanceWatcher: 0x55c1e6d82000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-10-11T11:19:03.792 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi111.stderr:2023-10-11T11:19:03.789+0000 7f2c226c4700 -1 rbd::mirror::InstanceWatcher: 0x55c1e690fc00 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-10-11T11:19:03.792 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi111.stderr:2023-10-11T11:19:03.789+0000 7f2c296d2700 -1 rbd::mirror::MirrorStatusUpdater 0x55c1e69125a0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-10-11T11:19:03.793 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi111.stderr:2023-10-11T11:19:03.789+0000 7f2c296d2700 -1 rbd::mirror::MirrorStatusUpdater 0x55c1e6912360 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-10-11T11:19:03.793 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi111.stderr:2023-10-11T11:19:03.790+0000 7f2c366ec700 -1 rbd::mirror::NamespaceReplayer: 0x55c1e68ba820 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-10-11T11:19:03.793 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi111.stderr:2023-10-11T11:19:03.790+0000 7f2c366ec700 -1 rbd::mirror::NamespaceReplayer: 0x55c1e68ba9c0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-10-11T11:19:03.793 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi111.stderr:2023-10-11T11:19:03.790+0000 7f2c306e0700 -1 rbd::mirror::MirrorStatusUpdater 0x55c1e6912000 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-10-11T11:19:03.793 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi111.stderr:2023-10-11T11:19:03.790+0000 7f2c306e0700 -1 rbd::mirror::MirrorStatusUpdater 0x55c1e6912120 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-10-11T11:19:03.794 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi111.stderr:2023-10-11T11:19:03.790+0000 7f2c36eed700 -1 rbd::mirror::NamespaceReplayer: 0x55c1e68ba820 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-10-11T11:19:03.794 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi111.stderr:2023-10-11T11:19:03.790+0000 7f2c36eed700 -1 rbd::mirror::NamespaceReplayer: 0x55c1e68ba9c0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-10-11T11:19:03.794 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi111.stderr:2023-10-11T11:19:03.790+0000 7f2c1beb7700 -1 rbd::mirror::MirrorStatusUpdater 0x55c1e69126c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-10-11T11:19:03.794 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi111.stderr:2023-10-11T11:19:03.790+0000 7f2c366ec700 -1 rbd::mirror::NamespaceReplayer: 0x55c1e6cb84e0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-10-11T11:19:03.794 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi111.stderr:2023-10-11T11:19:03.790+0000 7f2c226c4700 -1 rbd::mirror::MirrorStatusUpdater 0x55c1e6b405a0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-10-11T11:19:03.795 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi111.stderr:2023-10-11T11:19:03.790+0000 7f2c36eed700 -1 rbd::mirror::NamespaceReplayer: 0x55c1e6cb84e0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-10-11T11:19:03.796 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi111.stderr:2023-10-11T11:19:03.792+0000 7f2c366ec700 -1 rbd::mirror::LeaderWatcher: 0x55c1e6888f00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2023-10-11T11:19:03.796 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi111.stderr:2023-10-11T11:19:03.792+0000 7f2c366ec700 -1 rbd::mirror::InstanceWatcher: 0x55c1e66fee00 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.4886 object: (2) No such file or directory 2023-10-11T11:19:03.796 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi111.stderr:2023-10-11T11:19:03.793+0000 7f2c2fedf700 -1 rbd::mirror::InstanceWatcher: 0x55c1e66fee00 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-10-11T11:19:03.796 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi111.stderr:2023-10-11T11:19:03.793+0000 7f2c29ed3700 -1 rbd::mirror::MirrorStatusUpdater 0x55c1e6844360 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-10-11T11:19:03.796 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi111.stderr:2023-10-11T11:19:03.793+0000 7f2c366ec700 -1 rbd::mirror::NamespaceReplayer: 0x55c1e68ba000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-10-11T11:19:03.797 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi111.stderr:2023-10-11T11:19:03.793+0000 7f2c306e0700 -1 rbd::mirror::MirrorStatusUpdater 0x55c1e6672900 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-10-11T11:19:03.797 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi111.stderr:2023-10-11T11:19:03.793+0000 7f2c36eed700 -1 rbd::mirror::NamespaceReplayer: 0x55c1e68ba000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-10-11T11:19:03.806 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Stopped 2023-10-11T11:19:03.807 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2023-10-11T11:19:03.817 DEBUG:tasks.rbd_mirror.cluster1.client.mirror.0:waiting for process to exit 2023-10-11T11:19:03.817 INFO:teuthology.orchestra.run:waiting for 300 2023-10-11T11:19:03.848 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi111.stderr:2023-10-11T11:19:03.847+0000 7f25f7cea700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 142454) UID: 1000 2023-10-11T11:19:03.848 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi111.stderr:2023-10-11T11:19:03.847+0000 7f25f74e9700 -1 rbd::mirror::LeaderWatcher: 0x564a46134f00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2023-10-11T11:19:03.850 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi111.stderr:2023-10-11T11:19:03.848+0000 7f25f6ce8700 -1 rbd::mirror::InstanceWatcher: 0x564a45dcbc00 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6110 object: (2) No such file or directory 2023-10-11T11:19:03.850 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi111.stderr:2023-10-11T11:19:03.848+0000 7f25f6ce8700 -1 rbd::mirror::InstanceWatcher: 0x564a45c0d500 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6110 object: (2) No such file or directory 2023-10-11T11:19:03.850 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi111.stderr:2023-10-11T11:19:03.848+0000 7f25f74e9700 -1 rbd::mirror::InstanceWatcher: 0x564a45e48380 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6113 object: (2) No such file or directory 2023-10-11T11:19:03.851 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi111.stderr:2023-10-11T11:19:03.848+0000 7f25f0cdc700 -1 rbd::mirror::InstanceWatcher: 0x564a45dcbc00 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-10-11T11:19:03.851 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi111.stderr:2023-10-11T11:19:03.848+0000 7f25f0cdc700 -1 rbd::mirror::InstanceWatcher: 0x564a45c0d500 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-10-11T11:19:03.851 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi111.stderr:2023-10-11T11:19:03.848+0000 7f25e2cc0700 -1 rbd::mirror::InstanceWatcher: 0x564a45e48380 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-10-11T11:19:03.851 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi111.stderr:2023-10-11T11:19:03.848+0000 7f25e9cce700 -1 rbd::mirror::MirrorStatusUpdater 0x564a45e305a0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-10-11T11:19:03.851 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi111.stderr:2023-10-11T11:19:03.848+0000 7f25e9cce700 -1 rbd::mirror::MirrorStatusUpdater 0x564a45e30360 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-10-11T11:19:03.852 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi111.stderr:2023-10-11T11:19:03.848+0000 7f25f6ce8700 -1 rbd::mirror::NamespaceReplayer: 0x564a45dbe820 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-10-11T11:19:03.852 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi111.stderr:2023-10-11T11:19:03.848+0000 7f25f6ce8700 -1 rbd::mirror::NamespaceReplayer: 0x564a45dbe9c0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-10-11T11:19:03.852 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi111.stderr:2023-10-11T11:19:03.848+0000 7f25f04db700 -1 rbd::mirror::MirrorStatusUpdater 0x564a45e30000 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-10-11T11:19:03.852 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi111.stderr:2023-10-11T11:19:03.848+0000 7f25f04db700 -1 rbd::mirror::MirrorStatusUpdater 0x564a45e30120 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-10-11T11:19:03.852 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi111.stderr:2023-10-11T11:19:03.848+0000 7f25f74e9700 -1 rbd::mirror::NamespaceReplayer: 0x564a45dbe820 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-10-11T11:19:03.853 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi111.stderr:2023-10-11T11:19:03.848+0000 7f25f74e9700 -1 rbd::mirror::NamespaceReplayer: 0x564a45dbe9c0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-10-11T11:19:03.853 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi111.stderr:2023-10-11T11:19:03.849+0000 7f25dc4b3700 -1 rbd::mirror::MirrorStatusUpdater 0x564a4604e6c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-10-11T11:19:03.853 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi111.stderr:2023-10-11T11:19:03.849+0000 7f25f6ce8700 -1 rbd::mirror::NamespaceReplayer: 0x564a46122680 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-10-11T11:19:03.853 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi111.stderr:2023-10-11T11:19:03.849+0000 7f25e2cc0700 -1 rbd::mirror::MirrorStatusUpdater 0x564a46006240 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-10-11T11:19:03.853 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi111.stderr:2023-10-11T11:19:03.849+0000 7f25f74e9700 -1 rbd::mirror::NamespaceReplayer: 0x564a46122680 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-10-11T11:19:03.854 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi111.stderr:2023-10-11T11:19:03.851+0000 7f25f6ce8700 -1 rbd::mirror::LeaderWatcher: 0x564a45da4900 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2023-10-11T11:19:03.855 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi111.stderr:2023-10-11T11:19:03.851+0000 7f25f6ce8700 -1 rbd::mirror::InstanceWatcher: 0x564a45c0ce00 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6110 object: (2) No such file or directory 2023-10-11T11:19:03.855 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi111.stderr:2023-10-11T11:19:03.851+0000 7f25f04db700 -1 rbd::mirror::InstanceWatcher: 0x564a45c0ce00 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-10-11T11:19:03.855 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi111.stderr:2023-10-11T11:19:03.851+0000 7f25e9cce700 -1 rbd::mirror::MirrorStatusUpdater 0x564a45b726c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-10-11T11:19:03.855 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi111.stderr:2023-10-11T11:19:03.851+0000 7f25f6ce8700 -1 rbd::mirror::NamespaceReplayer: 0x564a45dbe000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-10-11T11:19:03.855 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi111.stderr:2023-10-11T11:19:03.852+0000 7f25f04db700 -1 rbd::mirror::MirrorStatusUpdater 0x564a4678c6c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-10-11T11:19:03.856 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi111.stderr:2023-10-11T11:19:03.852+0000 7f25f74e9700 -1 rbd::mirror::NamespaceReplayer: 0x564a45dbe000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-10-11T11:19:03.862 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Stopped 2023-10-11T11:19:03.863 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-10-11T11:19:03.876 INFO:tasks.ceph.ceph_manager.cluster2:waiting for clean 2023-10-11T11:19:03.876 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-10-11T11:19:04.219 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T11:19:04.222 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2023-10-11T11:19:04.233 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":1562,"stamp":"2023-10-11T11:19:02.766621+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":11,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":276,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3064228,"kb_used_data":418944,"kb_used_omap":292,"kb_used_meta":2640859,"kb_avail":280051292,"statfs":{"total":289910292480,"available":286772523008,"internally_reserved":0,"allocated":428998656,"data_stored":420423286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299441,"internal_metadata":2704240207},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536790253,"num_objects":-167,"num_object_clones":0,"num_object_copies":-334,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-123,"num_objects_misplaced":-9,"num_objects_unfound":0,"num_objects_dirty":-167,"num_whiteouts":0,"num_read":-139618,"num_read_kb":-39267355,"num_write":-411035,"num_write_kb":-147523554,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-5,"num_bytes_recovered":-8388608,"num_keys_recovered":-2,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.061056"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":514,"reported_epoch":497,"state":"active+undersized","last_fresh":"2023-10-11T11:18:53.702663+0000","last_change":"2023-10-11T11:15:31.661747+0000","last_active":"2023-10-11T11:18:53.702663+0000","last_peered":"2023-10-11T11:18:53.702663+0000","last_clean":"2023-10-11T11:15:27.638215+0000","last_became_active":"2023-10-11T11:15:31.661747+0000","last_became_peered":"2023-10-11T11:15:31.661747+0000","last_unstale":"2023-10-11T11:18:53.702663+0000","last_undegraded":"2023-10-11T11:18:53.702663+0000","last_fullsized":"2023-10-11T11:15:31.649336+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:15:25.047039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":26,"reported_epoch":497,"state":"active+undersized","last_fresh":"2023-10-11T11:18:53.700167+0000","last_change":"2023-10-11T11:15:31.671363+0000","last_active":"2023-10-11T11:18:53.700167+0000","last_peered":"2023-10-11T11:18:53.700167+0000","last_clean":"2023-10-11T10:43:40.751300+0000","last_became_active":"2023-10-11T11:15:31.671363+0000","last_became_peered":"2023-10-11T11:15:31.671363+0000","last_unstale":"2023-10-11T11:18:53.700167+0000","last_undegraded":"2023-10-11T11:18:53.700167+0000","last_fullsized":"2023-10-11T11:15:31.652001+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:01:53.884964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":509,"reported_epoch":497,"state":"active+clean","last_fresh":"2023-10-11T11:18:53.705545+0000","last_change":"2023-10-11T10:43:41.772781+0000","last_active":"2023-10-11T11:18:53.705545+0000","last_peered":"2023-10-11T11:18:53.705545+0000","last_clean":"2023-10-11T11:18:53.705545+0000","last_became_active":"2023-10-11T10:43:41.772443+0000","last_became_peered":"2023-10-11T10:43:41.772443+0000","last_unstale":"2023-10-11T11:18:53.705545+0000","last_undegraded":"2023-10-11T11:18:53.705545+0000","last_fullsized":"2023-10-11T11:18:53.705545+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T14:15:30.547284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":510,"reported_epoch":497,"state":"active+clean","last_fresh":"2023-10-11T11:18:53.700184+0000","last_change":"2023-10-11T10:43:41.772615+0000","last_active":"2023-10-11T11:18:53.700184+0000","last_peered":"2023-10-11T11:18:53.700184+0000","last_clean":"2023-10-11T11:18:53.700184+0000","last_became_active":"2023-10-11T10:43:41.772290+0000","last_became_peered":"2023-10-11T10:43:41.772290+0000","last_unstale":"2023-10-11T11:18:53.700184+0000","last_undegraded":"2023-10-11T11:18:53.700184+0000","last_fullsized":"2023-10-11T11:18:53.700184+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:26:06.909963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":645,"reported_epoch":497,"state":"active+undersized+degraded","last_fresh":"2023-10-11T11:18:53.698659+0000","last_change":"2023-10-11T11:15:31.654022+0000","last_active":"2023-10-11T11:18:53.698659+0000","last_peered":"2023-10-11T11:18:53.698659+0000","last_clean":"2023-10-11T11:15:27.636884+0000","last_became_active":"2023-10-11T11:15:31.654022+0000","last_became_peered":"2023-10-11T11:15:31.654022+0000","last_unstale":"2023-10-11T11:18:53.698659+0000","last_undegraded":"2023-10-11T11:15:31.646946+0000","last_fullsized":"2023-10-11T11:15:31.646751+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:36.739637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:36.739637+0000","last_clean_scrub_stamp":"2023-10-11T10:43:36.739637+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T11:33:33.696447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":517,"reported_epoch":497,"state":"active+clean+remapped","last_fresh":"2023-10-11T11:18:53.705572+0000","last_change":"2023-10-11T11:15:33.660200+0000","last_active":"2023-10-11T11:18:53.705572+0000","last_peered":"2023-10-11T11:18:53.705572+0000","last_clean":"2023-10-11T11:18:53.705572+0000","last_became_active":"2023-10-11T11:15:33.659619+0000","last_became_peered":"2023-10-11T11:15:33.659619+0000","last_unstale":"2023-10-11T11:18:53.705572+0000","last_undegraded":"2023-10-11T11:18:53.705572+0000","last_fullsized":"2023-10-11T11:18:53.705572+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T17:21:16.866392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":514,"reported_epoch":497,"state":"active+undersized","last_fresh":"2023-10-11T11:18:53.703415+0000","last_change":"2023-10-11T11:15:31.670813+0000","last_active":"2023-10-11T11:18:53.703415+0000","last_peered":"2023-10-11T11:18:53.703415+0000","last_clean":"2023-10-11T11:15:27.638219+0000","last_became_active":"2023-10-11T11:15:31.670813+0000","last_became_peered":"2023-10-11T11:15:31.670813+0000","last_unstale":"2023-10-11T11:18:53.703415+0000","last_undegraded":"2023-10-11T11:18:53.703415+0000","last_fullsized":"2023-10-11T11:15:31.658785+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:41:50.872346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":514,"reported_epoch":497,"state":"active+undersized","last_fresh":"2023-10-11T11:18:53.701612+0000","last_change":"2023-10-11T11:15:31.666378+0000","last_active":"2023-10-11T11:18:53.701612+0000","last_peered":"2023-10-11T11:18:53.701612+0000","last_clean":"2023-10-11T11:15:27.638303+0000","last_became_active":"2023-10-11T11:15:31.666378+0000","last_became_peered":"2023-10-11T11:15:31.666378+0000","last_unstale":"2023-10-11T11:18:53.701612+0000","last_undegraded":"2023-10-11T11:18:53.701612+0000","last_fullsized":"2023-10-11T11:15:31.649186+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T15:35:19.331209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":514,"reported_epoch":497,"state":"active+undersized","last_fresh":"2023-10-11T11:18:53.701564+0000","last_change":"2023-10-11T11:15:31.679465+0000","last_active":"2023-10-11T11:18:53.701564+0000","last_peered":"2023-10-11T11:18:53.701564+0000","last_clean":"2023-10-11T11:15:27.638749+0000","last_became_active":"2023-10-11T11:15:31.679465+0000","last_became_peered":"2023-10-11T11:15:31.679465+0000","last_unstale":"2023-10-11T11:18:53.701564+0000","last_undegraded":"2023-10-11T11:18:53.701564+0000","last_fullsized":"2023-10-11T11:15:31.659135+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T16:03:42.656692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":11,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738796,"num_pgs":83,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":789420,"kb_used_data":5788,"kb_used_omap":96,"kb_used_meta":779423,"kb_avail":93582420,"statfs":{"total":96636764160,"available":95828398080,"internally_reserved":0,"allocated":5926912,"data_stored":3092008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":98609,"internal_metadata":798129871},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 11:14:37 2023","interfaces":[{"interface":"back","average":{"1min":0.314,"5min":0.316,"15min":0.443},"min":{"1min":0.24399999999999999,"5min":0.14799999999999999,"15min":0.14599999999999999},"max":{"1min":0.48499999999999999,"5min":1.712,"15min":9.6400000000000006},"last":0.31},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.35499999999999998,"15min":0.41299999999999998},"min":{"1min":0.20899999999999999,"5min":0.17199999999999999,"15min":0.16400000000000001},"max":{"1min":3.8180000000000001,"5min":3.8180000000000001,"15min":6.3200000000000003},"last":0.26500000000000001}]},{"osd":1,"last update":"Wed Oct 11 11:18:39 2023","interfaces":[{"interface":"back","average":{"1min":0.27500000000000002,"5min":0.29299999999999998,"15min":0.32400000000000001},"min":{"1min":0.154,"5min":0.154,"15min":0.154},"max":{"1min":0.41999999999999998,"5min":0.59299999999999997,"15min":2.9540000000000002},"last":0.27900000000000003},{"interface":"front","average":{"1min":0.307,"5min":0.35899999999999999,"15min":0.36499999999999999},"min":{"1min":0.18099999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.45100000000000001,"5min":3.2719999999999998,"15min":3.2719999999999998},"last":0.24299999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738756,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1484496,"kb_used_data":406792,"kb_used_omap":107,"kb_used_meta":1077588,"kb_avail":92887344,"statfs":{"total":96636764160,"available":95116640256,"internally_reserved":0,"allocated":416555008,"data_stored":413648921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110309,"internal_metadata":1103450395},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Oct 11 11:14:30 2023","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.35399999999999998,"15min":0.40600000000000003},"min":{"1min":0.18099999999999999,"5min":0.155,"15min":0.14899999999999999},"max":{"1min":1.4319999999999999,"5min":2.3460000000000001,"15min":6.593},"last":0.26200000000000001},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.39100000000000001,"15min":0.48199999999999998},"min":{"1min":0.189,"5min":0.189,"15min":0.183},"max":{"1min":2.2120000000000002,"5min":2.4089999999999998,"15min":6.5960000000000001},"last":0.316}]},{"osd":2,"last update":"Wed Oct 11 11:14:30 2023","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.35499999999999998,"15min":0.47099999999999997},"min":{"1min":0.20300000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":1.2490000000000001,"5min":2.665,"15min":8.4730000000000008},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.375,"15min":0.442},"min":{"1min":0.23200000000000001,"5min":0.20499999999999999,"15min":0.19600000000000001},"max":{"1min":1.28,"5min":2.6230000000000002,"15min":7.0129999999999999},"last":0.33600000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738797,"num_pgs":98,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":790312,"kb_used_data":6364,"kb_used_omap":88,"kb_used_meta":783847,"kb_avail":93581528,"statfs":{"total":96636764160,"available":95827484672,"internally_reserved":0,"allocated":6516736,"data_stored":3682357,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":90523,"internal_metadata":802659941},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 11:14:33 2023","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.42799999999999999,"15min":0.41099999999999998},"min":{"1min":0.151,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":1.2809999999999999,"5min":3.5739999999999998,"15min":8.2479999999999993},"last":0.223},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.48299999999999998,"15min":0.49199999999999999},"min":{"1min":0.20699999999999999,"5min":0.182,"15min":0.17999999999999999},"max":{"1min":1.323,"5min":3.6080000000000001,"15min":8.2870000000000008},"last":0.28399999999999997}]},{"osd":2,"last update":"Wed Oct 11 11:18:40 2023","interfaces":[{"interface":"back","average":{"1min":0.33400000000000002,"5min":0.378,"15min":0.35599999999999998},"min":{"1min":0.16500000000000001,"5min":0.14699999999999999,"15min":0.14699999999999999},"max":{"1min":1.244,"5min":7.4829999999999997,"15min":10.617000000000001},"last":0.254},{"interface":"front","average":{"1min":0.34599999999999997,"5min":0.39000000000000001,"15min":0.34999999999999998},"min":{"1min":0.20399999999999999,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.81100000000000005,"5min":7.5350000000000001,"15min":7.5350000000000001},"last":0.223}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-11T11:19:04.234 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-10-11T11:19:04.550 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T11:19:04.553 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2023-10-11T11:19:04.563 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":1562,"stamp":"2023-10-11T11:19:02.766621+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":11,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":276,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3064228,"kb_used_data":418944,"kb_used_omap":292,"kb_used_meta":2640859,"kb_avail":280051292,"statfs":{"total":289910292480,"available":286772523008,"internally_reserved":0,"allocated":428998656,"data_stored":420423286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299441,"internal_metadata":2704240207},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536790253,"num_objects":-167,"num_object_clones":0,"num_object_copies":-334,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-123,"num_objects_misplaced":-9,"num_objects_unfound":0,"num_objects_dirty":-167,"num_whiteouts":0,"num_read":-139618,"num_read_kb":-39267355,"num_write":-411035,"num_write_kb":-147523554,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-5,"num_bytes_recovered":-8388608,"num_keys_recovered":-2,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.061056"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":514,"reported_epoch":497,"state":"active+undersized","last_fresh":"2023-10-11T11:18:53.702663+0000","last_change":"2023-10-11T11:15:31.661747+0000","last_active":"2023-10-11T11:18:53.702663+0000","last_peered":"2023-10-11T11:18:53.702663+0000","last_clean":"2023-10-11T11:15:27.638215+0000","last_became_active":"2023-10-11T11:15:31.661747+0000","last_became_peered":"2023-10-11T11:15:31.661747+0000","last_unstale":"2023-10-11T11:18:53.702663+0000","last_undegraded":"2023-10-11T11:18:53.702663+0000","last_fullsized":"2023-10-11T11:15:31.649336+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:15:25.047039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":26,"reported_epoch":497,"state":"active+undersized","last_fresh":"2023-10-11T11:18:53.700167+0000","last_change":"2023-10-11T11:15:31.671363+0000","last_active":"2023-10-11T11:18:53.700167+0000","last_peered":"2023-10-11T11:18:53.700167+0000","last_clean":"2023-10-11T10:43:40.751300+0000","last_became_active":"2023-10-11T11:15:31.671363+0000","last_became_peered":"2023-10-11T11:15:31.671363+0000","last_unstale":"2023-10-11T11:18:53.700167+0000","last_undegraded":"2023-10-11T11:18:53.700167+0000","last_fullsized":"2023-10-11T11:15:31.652001+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:01:53.884964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":509,"reported_epoch":497,"state":"active+clean","last_fresh":"2023-10-11T11:18:53.705545+0000","last_change":"2023-10-11T10:43:41.772781+0000","last_active":"2023-10-11T11:18:53.705545+0000","last_peered":"2023-10-11T11:18:53.705545+0000","last_clean":"2023-10-11T11:18:53.705545+0000","last_became_active":"2023-10-11T10:43:41.772443+0000","last_became_peered":"2023-10-11T10:43:41.772443+0000","last_unstale":"2023-10-11T11:18:53.705545+0000","last_undegraded":"2023-10-11T11:18:53.705545+0000","last_fullsized":"2023-10-11T11:18:53.705545+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T14:15:30.547284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":510,"reported_epoch":497,"state":"active+clean","last_fresh":"2023-10-11T11:18:53.700184+0000","last_change":"2023-10-11T10:43:41.772615+0000","last_active":"2023-10-11T11:18:53.700184+0000","last_peered":"2023-10-11T11:18:53.700184+0000","last_clean":"2023-10-11T11:18:53.700184+0000","last_became_active":"2023-10-11T10:43:41.772290+0000","last_became_peered":"2023-10-11T10:43:41.772290+0000","last_unstale":"2023-10-11T11:18:53.700184+0000","last_undegraded":"2023-10-11T11:18:53.700184+0000","last_fullsized":"2023-10-11T11:18:53.700184+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:26:06.909963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":645,"reported_epoch":497,"state":"active+undersized+degraded","last_fresh":"2023-10-11T11:18:53.698659+0000","last_change":"2023-10-11T11:15:31.654022+0000","last_active":"2023-10-11T11:18:53.698659+0000","last_peered":"2023-10-11T11:18:53.698659+0000","last_clean":"2023-10-11T11:15:27.636884+0000","last_became_active":"2023-10-11T11:15:31.654022+0000","last_became_peered":"2023-10-11T11:15:31.654022+0000","last_unstale":"2023-10-11T11:18:53.698659+0000","last_undegraded":"2023-10-11T11:15:31.646946+0000","last_fullsized":"2023-10-11T11:15:31.646751+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:36.739637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:36.739637+0000","last_clean_scrub_stamp":"2023-10-11T10:43:36.739637+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T11:33:33.696447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":517,"reported_epoch":497,"state":"active+clean+remapped","last_fresh":"2023-10-11T11:18:53.705572+0000","last_change":"2023-10-11T11:15:33.660200+0000","last_active":"2023-10-11T11:18:53.705572+0000","last_peered":"2023-10-11T11:18:53.705572+0000","last_clean":"2023-10-11T11:18:53.705572+0000","last_became_active":"2023-10-11T11:15:33.659619+0000","last_became_peered":"2023-10-11T11:15:33.659619+0000","last_unstale":"2023-10-11T11:18:53.705572+0000","last_undegraded":"2023-10-11T11:18:53.705572+0000","last_fullsized":"2023-10-11T11:18:53.705572+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T17:21:16.866392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":514,"reported_epoch":497,"state":"active+undersized","last_fresh":"2023-10-11T11:18:53.703415+0000","last_change":"2023-10-11T11:15:31.670813+0000","last_active":"2023-10-11T11:18:53.703415+0000","last_peered":"2023-10-11T11:18:53.703415+0000","last_clean":"2023-10-11T11:15:27.638219+0000","last_became_active":"2023-10-11T11:15:31.670813+0000","last_became_peered":"2023-10-11T11:15:31.670813+0000","last_unstale":"2023-10-11T11:18:53.703415+0000","last_undegraded":"2023-10-11T11:18:53.703415+0000","last_fullsized":"2023-10-11T11:15:31.658785+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:41:50.872346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":514,"reported_epoch":497,"state":"active+undersized","last_fresh":"2023-10-11T11:18:53.701612+0000","last_change":"2023-10-11T11:15:31.666378+0000","last_active":"2023-10-11T11:18:53.701612+0000","last_peered":"2023-10-11T11:18:53.701612+0000","last_clean":"2023-10-11T11:15:27.638303+0000","last_became_active":"2023-10-11T11:15:31.666378+0000","last_became_peered":"2023-10-11T11:15:31.666378+0000","last_unstale":"2023-10-11T11:18:53.701612+0000","last_undegraded":"2023-10-11T11:18:53.701612+0000","last_fullsized":"2023-10-11T11:15:31.649186+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T15:35:19.331209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":514,"reported_epoch":497,"state":"active+undersized","last_fresh":"2023-10-11T11:18:53.701564+0000","last_change":"2023-10-11T11:15:31.679465+0000","last_active":"2023-10-11T11:18:53.701564+0000","last_peered":"2023-10-11T11:18:53.701564+0000","last_clean":"2023-10-11T11:15:27.638749+0000","last_became_active":"2023-10-11T11:15:31.679465+0000","last_became_peered":"2023-10-11T11:15:31.679465+0000","last_unstale":"2023-10-11T11:18:53.701564+0000","last_undegraded":"2023-10-11T11:18:53.701564+0000","last_fullsized":"2023-10-11T11:15:31.659135+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T16:03:42.656692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":11,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738796,"num_pgs":83,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":789420,"kb_used_data":5788,"kb_used_omap":96,"kb_used_meta":779423,"kb_avail":93582420,"statfs":{"total":96636764160,"available":95828398080,"internally_reserved":0,"allocated":5926912,"data_stored":3092008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":98609,"internal_metadata":798129871},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 11:14:37 2023","interfaces":[{"interface":"back","average":{"1min":0.314,"5min":0.316,"15min":0.443},"min":{"1min":0.24399999999999999,"5min":0.14799999999999999,"15min":0.14599999999999999},"max":{"1min":0.48499999999999999,"5min":1.712,"15min":9.6400000000000006},"last":0.31},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.35499999999999998,"15min":0.41299999999999998},"min":{"1min":0.20899999999999999,"5min":0.17199999999999999,"15min":0.16400000000000001},"max":{"1min":3.8180000000000001,"5min":3.8180000000000001,"15min":6.3200000000000003},"last":0.26500000000000001}]},{"osd":1,"last update":"Wed Oct 11 11:18:39 2023","interfaces":[{"interface":"back","average":{"1min":0.27500000000000002,"5min":0.29299999999999998,"15min":0.32400000000000001},"min":{"1min":0.154,"5min":0.154,"15min":0.154},"max":{"1min":0.41999999999999998,"5min":0.59299999999999997,"15min":2.9540000000000002},"last":0.27900000000000003},{"interface":"front","average":{"1min":0.307,"5min":0.35899999999999999,"15min":0.36499999999999999},"min":{"1min":0.18099999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.45100000000000001,"5min":3.2719999999999998,"15min":3.2719999999999998},"last":0.24299999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738756,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1484496,"kb_used_data":406792,"kb_used_omap":107,"kb_used_meta":1077588,"kb_avail":92887344,"statfs":{"total":96636764160,"available":95116640256,"internally_reserved":0,"allocated":416555008,"data_stored":413648921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110309,"internal_metadata":1103450395},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Oct 11 11:14:30 2023","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.35399999999999998,"15min":0.40600000000000003},"min":{"1min":0.18099999999999999,"5min":0.155,"15min":0.14899999999999999},"max":{"1min":1.4319999999999999,"5min":2.3460000000000001,"15min":6.593},"last":0.26200000000000001},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.39100000000000001,"15min":0.48199999999999998},"min":{"1min":0.189,"5min":0.189,"15min":0.183},"max":{"1min":2.2120000000000002,"5min":2.4089999999999998,"15min":6.5960000000000001},"last":0.316}]},{"osd":2,"last update":"Wed Oct 11 11:14:30 2023","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.35499999999999998,"15min":0.47099999999999997},"min":{"1min":0.20300000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":1.2490000000000001,"5min":2.665,"15min":8.4730000000000008},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.375,"15min":0.442},"min":{"1min":0.23200000000000001,"5min":0.20499999999999999,"15min":0.19600000000000001},"max":{"1min":1.28,"5min":2.6230000000000002,"15min":7.0129999999999999},"last":0.33600000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738797,"num_pgs":98,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":790312,"kb_used_data":6364,"kb_used_omap":88,"kb_used_meta":783847,"kb_avail":93581528,"statfs":{"total":96636764160,"available":95827484672,"internally_reserved":0,"allocated":6516736,"data_stored":3682357,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":90523,"internal_metadata":802659941},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 11:14:33 2023","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.42799999999999999,"15min":0.41099999999999998},"min":{"1min":0.151,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":1.2809999999999999,"5min":3.5739999999999998,"15min":8.2479999999999993},"last":0.223},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.48299999999999998,"15min":0.49199999999999999},"min":{"1min":0.20699999999999999,"5min":0.182,"15min":0.17999999999999999},"max":{"1min":1.323,"5min":3.6080000000000001,"15min":8.2870000000000008},"last":0.28399999999999997}]},{"osd":2,"last update":"Wed Oct 11 11:18:40 2023","interfaces":[{"interface":"back","average":{"1min":0.33400000000000002,"5min":0.378,"15min":0.35599999999999998},"min":{"1min":0.16500000000000001,"5min":0.14699999999999999,"15min":0.14699999999999999},"max":{"1min":1.244,"5min":7.4829999999999997,"15min":10.617000000000001},"last":0.254},{"interface":"front","average":{"1min":0.34599999999999997,"5min":0.39000000000000001,"15min":0.34999999999999998},"min":{"1min":0.20399999999999999,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.81100000000000005,"5min":7.5350000000000001,"15min":7.5350000000000001},"last":0.223}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-11T11:19:04.564 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-10-11T11:19:04.882 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T11:19:04.885 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2023-10-11T11:19:04.896 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":1563,"stamp":"2023-10-11T11:19:04.644378+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":11,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":276,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3064228,"kb_used_data":418944,"kb_used_omap":292,"kb_used_meta":2640859,"kb_avail":280051292,"statfs":{"total":289910292480,"available":286772523008,"internally_reserved":0,"allocated":428998656,"data_stored":420423286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299441,"internal_metadata":2704240207},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-239075432,"num_objects":-93,"num_object_clones":0,"num_object_copies":-186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-65,"num_objects_misplaced":-6,"num_objects_unfound":0,"num_objects_dirty":-93,"num_whiteouts":0,"num_read":-139845,"num_read_kb":-39267555,"num_write":-411143,"num_write_kb":-147523563,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-5,"num_bytes_recovered":-8388608,"num_keys_recovered":-2,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002870"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":514,"reported_epoch":497,"state":"active+undersized","last_fresh":"2023-10-11T11:18:53.702663+0000","last_change":"2023-10-11T11:15:31.661747+0000","last_active":"2023-10-11T11:18:53.702663+0000","last_peered":"2023-10-11T11:18:53.702663+0000","last_clean":"2023-10-11T11:15:27.638215+0000","last_became_active":"2023-10-11T11:15:31.661747+0000","last_became_peered":"2023-10-11T11:15:31.661747+0000","last_unstale":"2023-10-11T11:18:53.702663+0000","last_undegraded":"2023-10-11T11:18:53.702663+0000","last_fullsized":"2023-10-11T11:15:31.649336+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:15:25.047039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":26,"reported_epoch":497,"state":"active+undersized","last_fresh":"2023-10-11T11:18:53.700167+0000","last_change":"2023-10-11T11:15:31.671363+0000","last_active":"2023-10-11T11:18:53.700167+0000","last_peered":"2023-10-11T11:18:53.700167+0000","last_clean":"2023-10-11T10:43:40.751300+0000","last_became_active":"2023-10-11T11:15:31.671363+0000","last_became_peered":"2023-10-11T11:15:31.671363+0000","last_unstale":"2023-10-11T11:18:53.700167+0000","last_undegraded":"2023-10-11T11:18:53.700167+0000","last_fullsized":"2023-10-11T11:15:31.652001+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:01:53.884964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":509,"reported_epoch":497,"state":"active+clean","last_fresh":"2023-10-11T11:18:53.705545+0000","last_change":"2023-10-11T10:43:41.772781+0000","last_active":"2023-10-11T11:18:53.705545+0000","last_peered":"2023-10-11T11:18:53.705545+0000","last_clean":"2023-10-11T11:18:53.705545+0000","last_became_active":"2023-10-11T10:43:41.772443+0000","last_became_peered":"2023-10-11T10:43:41.772443+0000","last_unstale":"2023-10-11T11:18:53.705545+0000","last_undegraded":"2023-10-11T11:18:53.705545+0000","last_fullsized":"2023-10-11T11:18:53.705545+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T14:15:30.547284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":510,"reported_epoch":497,"state":"active+clean","last_fresh":"2023-10-11T11:18:53.700184+0000","last_change":"2023-10-11T10:43:41.772615+0000","last_active":"2023-10-11T11:18:53.700184+0000","last_peered":"2023-10-11T11:18:53.700184+0000","last_clean":"2023-10-11T11:18:53.700184+0000","last_became_active":"2023-10-11T10:43:41.772290+0000","last_became_peered":"2023-10-11T10:43:41.772290+0000","last_unstale":"2023-10-11T11:18:53.700184+0000","last_undegraded":"2023-10-11T11:18:53.700184+0000","last_fullsized":"2023-10-11T11:18:53.700184+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:26:06.909963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":645,"reported_epoch":497,"state":"active+undersized+degraded","last_fresh":"2023-10-11T11:18:53.698659+0000","last_change":"2023-10-11T11:15:31.654022+0000","last_active":"2023-10-11T11:18:53.698659+0000","last_peered":"2023-10-11T11:18:53.698659+0000","last_clean":"2023-10-11T11:15:27.636884+0000","last_became_active":"2023-10-11T11:15:31.654022+0000","last_became_peered":"2023-10-11T11:15:31.654022+0000","last_unstale":"2023-10-11T11:18:53.698659+0000","last_undegraded":"2023-10-11T11:15:31.646946+0000","last_fullsized":"2023-10-11T11:15:31.646751+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:36.739637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:36.739637+0000","last_clean_scrub_stamp":"2023-10-11T10:43:36.739637+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T11:33:33.696447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":517,"reported_epoch":497,"state":"active+clean+remapped","last_fresh":"2023-10-11T11:18:53.705572+0000","last_change":"2023-10-11T11:15:33.660200+0000","last_active":"2023-10-11T11:18:53.705572+0000","last_peered":"2023-10-11T11:18:53.705572+0000","last_clean":"2023-10-11T11:18:53.705572+0000","last_became_active":"2023-10-11T11:15:33.659619+0000","last_became_peered":"2023-10-11T11:15:33.659619+0000","last_unstale":"2023-10-11T11:18:53.705572+0000","last_undegraded":"2023-10-11T11:18:53.705572+0000","last_fullsized":"2023-10-11T11:18:53.705572+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T17:21:16.866392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":514,"reported_epoch":497,"state":"active+undersized","last_fresh":"2023-10-11T11:18:53.703415+0000","last_change":"2023-10-11T11:15:31.670813+0000","last_active":"2023-10-11T11:18:53.703415+0000","last_peered":"2023-10-11T11:18:53.703415+0000","last_clean":"2023-10-11T11:15:27.638219+0000","last_became_active":"2023-10-11T11:15:31.670813+0000","last_became_peered":"2023-10-11T11:15:31.670813+0000","last_unstale":"2023-10-11T11:18:53.703415+0000","last_undegraded":"2023-10-11T11:18:53.703415+0000","last_fullsized":"2023-10-11T11:15:31.658785+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:41:50.872346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":514,"reported_epoch":497,"state":"active+undersized","last_fresh":"2023-10-11T11:18:53.701612+0000","last_change":"2023-10-11T11:15:31.666378+0000","last_active":"2023-10-11T11:18:53.701612+0000","last_peered":"2023-10-11T11:18:53.701612+0000","last_clean":"2023-10-11T11:15:27.638303+0000","last_became_active":"2023-10-11T11:15:31.666378+0000","last_became_peered":"2023-10-11T11:15:31.666378+0000","last_unstale":"2023-10-11T11:18:53.701612+0000","last_undegraded":"2023-10-11T11:18:53.701612+0000","last_fullsized":"2023-10-11T11:15:31.649186+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T15:35:19.331209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":514,"reported_epoch":497,"state":"active+undersized","last_fresh":"2023-10-11T11:18:53.701564+0000","last_change":"2023-10-11T11:15:31.679465+0000","last_active":"2023-10-11T11:18:53.701564+0000","last_peered":"2023-10-11T11:18:53.701564+0000","last_clean":"2023-10-11T11:15:27.638749+0000","last_became_active":"2023-10-11T11:15:31.679465+0000","last_became_peered":"2023-10-11T11:15:31.679465+0000","last_unstale":"2023-10-11T11:18:53.701564+0000","last_undegraded":"2023-10-11T11:18:53.701564+0000","last_fullsized":"2023-10-11T11:15:31.659135+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T16:03:42.656692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":11,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738796,"num_pgs":83,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":789420,"kb_used_data":5788,"kb_used_omap":96,"kb_used_meta":779423,"kb_avail":93582420,"statfs":{"total":96636764160,"available":95828398080,"internally_reserved":0,"allocated":5926912,"data_stored":3092008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":98609,"internal_metadata":798129871},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 11:14:37 2023","interfaces":[{"interface":"back","average":{"1min":0.314,"5min":0.316,"15min":0.443},"min":{"1min":0.24399999999999999,"5min":0.14799999999999999,"15min":0.14599999999999999},"max":{"1min":0.48499999999999999,"5min":1.712,"15min":9.6400000000000006},"last":0.31},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.35499999999999998,"15min":0.41299999999999998},"min":{"1min":0.20899999999999999,"5min":0.17199999999999999,"15min":0.16400000000000001},"max":{"1min":3.8180000000000001,"5min":3.8180000000000001,"15min":6.3200000000000003},"last":0.26500000000000001}]},{"osd":1,"last update":"Wed Oct 11 11:18:39 2023","interfaces":[{"interface":"back","average":{"1min":0.27500000000000002,"5min":0.29299999999999998,"15min":0.32400000000000001},"min":{"1min":0.154,"5min":0.154,"15min":0.154},"max":{"1min":0.41999999999999998,"5min":0.59299999999999997,"15min":2.9540000000000002},"last":0.27900000000000003},{"interface":"front","average":{"1min":0.307,"5min":0.35899999999999999,"15min":0.36499999999999999},"min":{"1min":0.18099999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.45100000000000001,"5min":3.2719999999999998,"15min":3.2719999999999998},"last":0.24299999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738756,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1484496,"kb_used_data":406792,"kb_used_omap":107,"kb_used_meta":1077588,"kb_avail":92887344,"statfs":{"total":96636764160,"available":95116640256,"internally_reserved":0,"allocated":416555008,"data_stored":413648921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110309,"internal_metadata":1103450395},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Oct 11 11:14:30 2023","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.35399999999999998,"15min":0.40600000000000003},"min":{"1min":0.18099999999999999,"5min":0.155,"15min":0.14899999999999999},"max":{"1min":1.4319999999999999,"5min":2.3460000000000001,"15min":6.593},"last":0.26200000000000001},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.39100000000000001,"15min":0.48199999999999998},"min":{"1min":0.189,"5min":0.189,"15min":0.183},"max":{"1min":2.2120000000000002,"5min":2.4089999999999998,"15min":6.5960000000000001},"last":0.316}]},{"osd":2,"last update":"Wed Oct 11 11:14:30 2023","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.35499999999999998,"15min":0.47099999999999997},"min":{"1min":0.20300000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":1.2490000000000001,"5min":2.665,"15min":8.4730000000000008},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.375,"15min":0.442},"min":{"1min":0.23200000000000001,"5min":0.20499999999999999,"15min":0.19600000000000001},"max":{"1min":1.28,"5min":2.6230000000000002,"15min":7.0129999999999999},"last":0.33600000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738797,"num_pgs":98,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":790312,"kb_used_data":6364,"kb_used_omap":88,"kb_used_meta":783847,"kb_avail":93581528,"statfs":{"total":96636764160,"available":95827484672,"internally_reserved":0,"allocated":6516736,"data_stored":3682357,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":90523,"internal_metadata":802659941},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 11:14:33 2023","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.42799999999999999,"15min":0.41099999999999998},"min":{"1min":0.151,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":1.2809999999999999,"5min":3.5739999999999998,"15min":8.2479999999999993},"last":0.223},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.48299999999999998,"15min":0.49199999999999999},"min":{"1min":0.20699999999999999,"5min":0.182,"15min":0.17999999999999999},"max":{"1min":1.323,"5min":3.6080000000000001,"15min":8.2870000000000008},"last":0.28399999999999997}]},{"osd":2,"last update":"Wed Oct 11 11:18:40 2023","interfaces":[{"interface":"back","average":{"1min":0.33400000000000002,"5min":0.378,"15min":0.35599999999999998},"min":{"1min":0.16500000000000001,"5min":0.14699999999999999,"15min":0.14699999999999999},"max":{"1min":1.244,"5min":7.4829999999999997,"15min":10.617000000000001},"last":0.254},{"interface":"front","average":{"1min":0.34599999999999997,"5min":0.39000000000000001,"15min":0.34999999999999998},"min":{"1min":0.20399999999999999,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.81100000000000005,"5min":7.5350000000000001,"15min":7.5350000000000001},"last":0.223}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-11T11:19:04.897 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-10-11T11:19:04.898 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 514, 'reported_epoch': 497, 'state': 'active+undersized', 'last_fresh': '2023-10-11T11:18:53.702663+0000', 'last_change': '2023-10-11T11:15:31.661747+0000', 'last_active': '2023-10-11T11:18:53.702663+0000', 'last_peered': '2023-10-11T11:18:53.702663+0000', 'last_clean': '2023-10-11T11:15:27.638215+0000', 'last_became_active': '2023-10-11T11:15:31.661747+0000', 'last_became_peered': '2023-10-11T11:15:31.661747+0000', 'last_unstale': '2023-10-11T11:18:53.702663+0000', 'last_undegraded': '2023-10-11T11:18:53.702663+0000', 'last_fullsized': '2023-10-11T11:15:31.649336+0000', 'mapping_epoch': 480, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_clean_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-12T22:15:25.047039+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-10-11T11:19:04.898 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-10-11T11:19:04.898 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 26, 'reported_epoch': 497, 'state': 'active+undersized', 'last_fresh': '2023-10-11T11:18:53.700167+0000', 'last_change': '2023-10-11T11:15:31.671363+0000', 'last_active': '2023-10-11T11:18:53.700167+0000', 'last_peered': '2023-10-11T11:18:53.700167+0000', 'last_clean': '2023-10-11T10:43:40.751300+0000', 'last_became_active': '2023-10-11T11:15:31.671363+0000', 'last_became_peered': '2023-10-11T11:15:31.671363+0000', 'last_unstale': '2023-10-11T11:18:53.700167+0000', 'last_undegraded': '2023-10-11T11:18:53.700167+0000', 'last_fullsized': '2023-10-11T11:15:31.652001+0000', 'mapping_epoch': 480, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_clean_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-12T21:01:53.884964+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-10-11T11:19:04.898 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-10-11T11:19:04.899 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "11'97", 'reported_seq': 645, 'reported_epoch': 497, 'state': 'active+undersized+degraded', 'last_fresh': '2023-10-11T11:18:53.698659+0000', 'last_change': '2023-10-11T11:15:31.654022+0000', 'last_active': '2023-10-11T11:18:53.698659+0000', 'last_peered': '2023-10-11T11:18:53.698659+0000', 'last_clean': '2023-10-11T11:15:27.636884+0000', 'last_became_active': '2023-10-11T11:15:31.654022+0000', 'last_became_peered': '2023-10-11T11:15:31.654022+0000', 'last_unstale': '2023-10-11T11:18:53.698659+0000', 'last_undegraded': '2023-10-11T11:15:31.646946+0000', 'last_fullsized': '2023-10-11T11:15:31.646751+0000', 'mapping_epoch': 480, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-11T10:43:36.739637+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-11T10:43:36.739637+0000', 'last_clean_scrub_stamp': '2023-10-11T10:43:36.739637+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-12T11:33:33.696447+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-10-11T11:19:04.899 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.0 is not active+clean 2023-10-11T11:19:04.899 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 517, 'reported_epoch': 497, 'state': 'active+clean+remapped', 'last_fresh': '2023-10-11T11:18:53.705572+0000', 'last_change': '2023-10-11T11:15:33.660200+0000', 'last_active': '2023-10-11T11:18:53.705572+0000', 'last_peered': '2023-10-11T11:18:53.705572+0000', 'last_clean': '2023-10-11T11:18:53.705572+0000', 'last_became_active': '2023-10-11T11:15:33.659619+0000', 'last_became_peered': '2023-10-11T11:15:33.659619+0000', 'last_unstale': '2023-10-11T11:18:53.705572+0000', 'last_undegraded': '2023-10-11T11:18:53.705572+0000', 'last_fullsized': '2023-10-11T11:18:53.705572+0000', 'mapping_epoch': 482, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 483, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_clean_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-12T17:21:16.866392+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2, 1], 'avail_no_missing': ['2', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-10-11T11:19:04.900 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-10-11T11:19:04.900 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 514, 'reported_epoch': 497, 'state': 'active+undersized', 'last_fresh': '2023-10-11T11:18:53.703415+0000', 'last_change': '2023-10-11T11:15:31.670813+0000', 'last_active': '2023-10-11T11:18:53.703415+0000', 'last_peered': '2023-10-11T11:18:53.703415+0000', 'last_clean': '2023-10-11T11:15:27.638219+0000', 'last_became_active': '2023-10-11T11:15:31.670813+0000', 'last_became_peered': '2023-10-11T11:15:31.670813+0000', 'last_unstale': '2023-10-11T11:18:53.703415+0000', 'last_undegraded': '2023-10-11T11:18:53.703415+0000', 'last_fullsized': '2023-10-11T11:15:31.658785+0000', 'mapping_epoch': 480, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_clean_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-12T21:41:50.872346+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-10-11T11:19:04.900 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-10-11T11:19:04.900 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 514, 'reported_epoch': 497, 'state': 'active+undersized', 'last_fresh': '2023-10-11T11:18:53.701612+0000', 'last_change': '2023-10-11T11:15:31.666378+0000', 'last_active': '2023-10-11T11:18:53.701612+0000', 'last_peered': '2023-10-11T11:18:53.701612+0000', 'last_clean': '2023-10-11T11:15:27.638303+0000', 'last_became_active': '2023-10-11T11:15:31.666378+0000', 'last_became_peered': '2023-10-11T11:15:31.666378+0000', 'last_unstale': '2023-10-11T11:18:53.701612+0000', 'last_undegraded': '2023-10-11T11:18:53.701612+0000', 'last_fullsized': '2023-10-11T11:15:31.649186+0000', 'mapping_epoch': 480, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_clean_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-12T15:35:19.331209+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-10-11T11:19:04.901 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-10-11T11:19:04.901 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 514, 'reported_epoch': 497, 'state': 'active+undersized', 'last_fresh': '2023-10-11T11:18:53.701564+0000', 'last_change': '2023-10-11T11:15:31.679465+0000', 'last_active': '2023-10-11T11:18:53.701564+0000', 'last_peered': '2023-10-11T11:18:53.701564+0000', 'last_clean': '2023-10-11T11:15:27.638749+0000', 'last_became_active': '2023-10-11T11:15:31.679465+0000', 'last_became_peered': '2023-10-11T11:15:31.679465+0000', 'last_unstale': '2023-10-11T11:18:53.701564+0000', 'last_undegraded': '2023-10-11T11:18:53.701564+0000', 'last_fullsized': '2023-10-11T11:15:31.659135+0000', 'mapping_epoch': 480, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_clean_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-12T16:03:42.656692+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-10-11T11:19:04.902 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-10-11T11:19:05.301 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T11:19:05.301 INFO:teuthology.orchestra.run.smithi111.stdout:{"fsid":"022ce6f2-9a5b-45ad-8b77-a1332d92f228","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2140,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":500,"num_osds":3,"num_up_osds":2,"osd_up_since":1697022930,"num_in_osds":3,"osd_in_since":1697021005,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":2},{"state_name":"active+undersized+degraded","count":1},{"state_name":"active+clean+remapped","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3137769472,"bytes_avail":286772523008,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":36,"modified":"2023-10-11T11:19:04.644408+0000","services":{}},"progress_events":{}} 2023-10-11T11:19:05.315 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-10-11T11:19:05.315 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-10-11T11:19:05.630 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T11:19:05.632 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2023-10-11T11:19:05.643 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":1563,"stamp":"2023-10-11T11:19:04.644378+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":11,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":276,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3064228,"kb_used_data":418944,"kb_used_omap":292,"kb_used_meta":2640859,"kb_avail":280051292,"statfs":{"total":289910292480,"available":286772523008,"internally_reserved":0,"allocated":428998656,"data_stored":420423286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299441,"internal_metadata":2704240207},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-239075432,"num_objects":-93,"num_object_clones":0,"num_object_copies":-186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-65,"num_objects_misplaced":-6,"num_objects_unfound":0,"num_objects_dirty":-93,"num_whiteouts":0,"num_read":-139845,"num_read_kb":-39267555,"num_write":-411143,"num_write_kb":-147523563,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-5,"num_bytes_recovered":-8388608,"num_keys_recovered":-2,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002870"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":514,"reported_epoch":497,"state":"active+undersized","last_fresh":"2023-10-11T11:18:53.702663+0000","last_change":"2023-10-11T11:15:31.661747+0000","last_active":"2023-10-11T11:18:53.702663+0000","last_peered":"2023-10-11T11:18:53.702663+0000","last_clean":"2023-10-11T11:15:27.638215+0000","last_became_active":"2023-10-11T11:15:31.661747+0000","last_became_peered":"2023-10-11T11:15:31.661747+0000","last_unstale":"2023-10-11T11:18:53.702663+0000","last_undegraded":"2023-10-11T11:18:53.702663+0000","last_fullsized":"2023-10-11T11:15:31.649336+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:15:25.047039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":26,"reported_epoch":497,"state":"active+undersized","last_fresh":"2023-10-11T11:18:53.700167+0000","last_change":"2023-10-11T11:15:31.671363+0000","last_active":"2023-10-11T11:18:53.700167+0000","last_peered":"2023-10-11T11:18:53.700167+0000","last_clean":"2023-10-11T10:43:40.751300+0000","last_became_active":"2023-10-11T11:15:31.671363+0000","last_became_peered":"2023-10-11T11:15:31.671363+0000","last_unstale":"2023-10-11T11:18:53.700167+0000","last_undegraded":"2023-10-11T11:18:53.700167+0000","last_fullsized":"2023-10-11T11:15:31.652001+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:01:53.884964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":509,"reported_epoch":497,"state":"active+clean","last_fresh":"2023-10-11T11:18:53.705545+0000","last_change":"2023-10-11T10:43:41.772781+0000","last_active":"2023-10-11T11:18:53.705545+0000","last_peered":"2023-10-11T11:18:53.705545+0000","last_clean":"2023-10-11T11:18:53.705545+0000","last_became_active":"2023-10-11T10:43:41.772443+0000","last_became_peered":"2023-10-11T10:43:41.772443+0000","last_unstale":"2023-10-11T11:18:53.705545+0000","last_undegraded":"2023-10-11T11:18:53.705545+0000","last_fullsized":"2023-10-11T11:18:53.705545+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T14:15:30.547284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":510,"reported_epoch":497,"state":"active+clean","last_fresh":"2023-10-11T11:18:53.700184+0000","last_change":"2023-10-11T10:43:41.772615+0000","last_active":"2023-10-11T11:18:53.700184+0000","last_peered":"2023-10-11T11:18:53.700184+0000","last_clean":"2023-10-11T11:18:53.700184+0000","last_became_active":"2023-10-11T10:43:41.772290+0000","last_became_peered":"2023-10-11T10:43:41.772290+0000","last_unstale":"2023-10-11T11:18:53.700184+0000","last_undegraded":"2023-10-11T11:18:53.700184+0000","last_fullsized":"2023-10-11T11:18:53.700184+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:26:06.909963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":645,"reported_epoch":497,"state":"active+undersized+degraded","last_fresh":"2023-10-11T11:18:53.698659+0000","last_change":"2023-10-11T11:15:31.654022+0000","last_active":"2023-10-11T11:18:53.698659+0000","last_peered":"2023-10-11T11:18:53.698659+0000","last_clean":"2023-10-11T11:15:27.636884+0000","last_became_active":"2023-10-11T11:15:31.654022+0000","last_became_peered":"2023-10-11T11:15:31.654022+0000","last_unstale":"2023-10-11T11:18:53.698659+0000","last_undegraded":"2023-10-11T11:15:31.646946+0000","last_fullsized":"2023-10-11T11:15:31.646751+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:36.739637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:36.739637+0000","last_clean_scrub_stamp":"2023-10-11T10:43:36.739637+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T11:33:33.696447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":517,"reported_epoch":497,"state":"active+clean+remapped","last_fresh":"2023-10-11T11:18:53.705572+0000","last_change":"2023-10-11T11:15:33.660200+0000","last_active":"2023-10-11T11:18:53.705572+0000","last_peered":"2023-10-11T11:18:53.705572+0000","last_clean":"2023-10-11T11:18:53.705572+0000","last_became_active":"2023-10-11T11:15:33.659619+0000","last_became_peered":"2023-10-11T11:15:33.659619+0000","last_unstale":"2023-10-11T11:18:53.705572+0000","last_undegraded":"2023-10-11T11:18:53.705572+0000","last_fullsized":"2023-10-11T11:18:53.705572+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T17:21:16.866392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":514,"reported_epoch":497,"state":"active+undersized","last_fresh":"2023-10-11T11:18:53.703415+0000","last_change":"2023-10-11T11:15:31.670813+0000","last_active":"2023-10-11T11:18:53.703415+0000","last_peered":"2023-10-11T11:18:53.703415+0000","last_clean":"2023-10-11T11:15:27.638219+0000","last_became_active":"2023-10-11T11:15:31.670813+0000","last_became_peered":"2023-10-11T11:15:31.670813+0000","last_unstale":"2023-10-11T11:18:53.703415+0000","last_undegraded":"2023-10-11T11:18:53.703415+0000","last_fullsized":"2023-10-11T11:15:31.658785+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:41:50.872346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":514,"reported_epoch":497,"state":"active+undersized","last_fresh":"2023-10-11T11:18:53.701612+0000","last_change":"2023-10-11T11:15:31.666378+0000","last_active":"2023-10-11T11:18:53.701612+0000","last_peered":"2023-10-11T11:18:53.701612+0000","last_clean":"2023-10-11T11:15:27.638303+0000","last_became_active":"2023-10-11T11:15:31.666378+0000","last_became_peered":"2023-10-11T11:15:31.666378+0000","last_unstale":"2023-10-11T11:18:53.701612+0000","last_undegraded":"2023-10-11T11:18:53.701612+0000","last_fullsized":"2023-10-11T11:15:31.649186+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T15:35:19.331209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":514,"reported_epoch":497,"state":"active+undersized","last_fresh":"2023-10-11T11:18:53.701564+0000","last_change":"2023-10-11T11:15:31.679465+0000","last_active":"2023-10-11T11:18:53.701564+0000","last_peered":"2023-10-11T11:18:53.701564+0000","last_clean":"2023-10-11T11:15:27.638749+0000","last_became_active":"2023-10-11T11:15:31.679465+0000","last_became_peered":"2023-10-11T11:15:31.679465+0000","last_unstale":"2023-10-11T11:18:53.701564+0000","last_undegraded":"2023-10-11T11:18:53.701564+0000","last_fullsized":"2023-10-11T11:15:31.659135+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T16:03:42.656692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":11,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738796,"num_pgs":83,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":789420,"kb_used_data":5788,"kb_used_omap":96,"kb_used_meta":779423,"kb_avail":93582420,"statfs":{"total":96636764160,"available":95828398080,"internally_reserved":0,"allocated":5926912,"data_stored":3092008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":98609,"internal_metadata":798129871},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 11:14:37 2023","interfaces":[{"interface":"back","average":{"1min":0.314,"5min":0.316,"15min":0.443},"min":{"1min":0.24399999999999999,"5min":0.14799999999999999,"15min":0.14599999999999999},"max":{"1min":0.48499999999999999,"5min":1.712,"15min":9.6400000000000006},"last":0.31},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.35499999999999998,"15min":0.41299999999999998},"min":{"1min":0.20899999999999999,"5min":0.17199999999999999,"15min":0.16400000000000001},"max":{"1min":3.8180000000000001,"5min":3.8180000000000001,"15min":6.3200000000000003},"last":0.26500000000000001}]},{"osd":1,"last update":"Wed Oct 11 11:18:39 2023","interfaces":[{"interface":"back","average":{"1min":0.27500000000000002,"5min":0.29299999999999998,"15min":0.32400000000000001},"min":{"1min":0.154,"5min":0.154,"15min":0.154},"max":{"1min":0.41999999999999998,"5min":0.59299999999999997,"15min":2.9540000000000002},"last":0.27900000000000003},{"interface":"front","average":{"1min":0.307,"5min":0.35899999999999999,"15min":0.36499999999999999},"min":{"1min":0.18099999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.45100000000000001,"5min":3.2719999999999998,"15min":3.2719999999999998},"last":0.24299999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738756,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1484496,"kb_used_data":406792,"kb_used_omap":107,"kb_used_meta":1077588,"kb_avail":92887344,"statfs":{"total":96636764160,"available":95116640256,"internally_reserved":0,"allocated":416555008,"data_stored":413648921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110309,"internal_metadata":1103450395},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Oct 11 11:14:30 2023","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.35399999999999998,"15min":0.40600000000000003},"min":{"1min":0.18099999999999999,"5min":0.155,"15min":0.14899999999999999},"max":{"1min":1.4319999999999999,"5min":2.3460000000000001,"15min":6.593},"last":0.26200000000000001},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.39100000000000001,"15min":0.48199999999999998},"min":{"1min":0.189,"5min":0.189,"15min":0.183},"max":{"1min":2.2120000000000002,"5min":2.4089999999999998,"15min":6.5960000000000001},"last":0.316}]},{"osd":2,"last update":"Wed Oct 11 11:14:30 2023","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.35499999999999998,"15min":0.47099999999999997},"min":{"1min":0.20300000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":1.2490000000000001,"5min":2.665,"15min":8.4730000000000008},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.375,"15min":0.442},"min":{"1min":0.23200000000000001,"5min":0.20499999999999999,"15min":0.19600000000000001},"max":{"1min":1.28,"5min":2.6230000000000002,"15min":7.0129999999999999},"last":0.33600000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738797,"num_pgs":98,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":790312,"kb_used_data":6364,"kb_used_omap":88,"kb_used_meta":783847,"kb_avail":93581528,"statfs":{"total":96636764160,"available":95827484672,"internally_reserved":0,"allocated":6516736,"data_stored":3682357,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":90523,"internal_metadata":802659941},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 11:14:33 2023","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.42799999999999999,"15min":0.41099999999999998},"min":{"1min":0.151,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":1.2809999999999999,"5min":3.5739999999999998,"15min":8.2479999999999993},"last":0.223},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.48299999999999998,"15min":0.49199999999999999},"min":{"1min":0.20699999999999999,"5min":0.182,"15min":0.17999999999999999},"max":{"1min":1.323,"5min":3.6080000000000001,"15min":8.2870000000000008},"last":0.28399999999999997}]},{"osd":2,"last update":"Wed Oct 11 11:18:40 2023","interfaces":[{"interface":"back","average":{"1min":0.33400000000000002,"5min":0.378,"15min":0.35599999999999998},"min":{"1min":0.16500000000000001,"5min":0.14699999999999999,"15min":0.14699999999999999},"max":{"1min":1.244,"5min":7.4829999999999997,"15min":10.617000000000001},"last":0.254},{"interface":"front","average":{"1min":0.34599999999999997,"5min":0.39000000000000001,"15min":0.34999999999999998},"min":{"1min":0.20399999999999999,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.81100000000000005,"5min":7.5350000000000001,"15min":7.5350000000000001},"last":0.223}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-11T11:19:08.313 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~216s 2023-10-11T11:19:08.646 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-10-11T11:19:08.975 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T11:19:08.977 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2023-10-11T11:19:08.989 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":1565,"stamp":"2023-10-11T11:19:08.645514+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":11,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":148,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3096284,"kb_used_data":418996,"kb_used_omap":292,"kb_used_meta":2676955,"kb_avail":280019236,"statfs":{"total":289910292480,"available":286739697664,"internally_reserved":0,"allocated":429051904,"data_stored":420453125,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299441,"internal_metadata":2741202511},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19,"num_objects":-8,"num_object_clones":0,"num_object_copies":-16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-8,"num_whiteouts":0,"num_read":-2040,"num_read_kb":-1943,"num_write":-383,"num_write_kb":-303,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.023021"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":516,"reported_epoch":499,"state":"active+undersized","last_fresh":"2023-10-11T11:19:00.633571+0000","last_change":"2023-10-11T11:15:31.661747+0000","last_active":"2023-10-11T11:19:00.633571+0000","last_peered":"2023-10-11T11:19:00.633571+0000","last_clean":"2023-10-11T11:15:27.638215+0000","last_became_active":"2023-10-11T11:15:31.661747+0000","last_became_peered":"2023-10-11T11:15:31.661747+0000","last_unstale":"2023-10-11T11:19:00.633571+0000","last_undegraded":"2023-10-11T11:19:00.633571+0000","last_fullsized":"2023-10-11T11:15:31.649336+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:15:25.047039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":28,"reported_epoch":499,"state":"active+undersized","last_fresh":"2023-10-11T11:19:00.628569+0000","last_change":"2023-10-11T11:15:31.671363+0000","last_active":"2023-10-11T11:19:00.628569+0000","last_peered":"2023-10-11T11:19:00.628569+0000","last_clean":"2023-10-11T10:43:40.751300+0000","last_became_active":"2023-10-11T11:15:31.671363+0000","last_became_peered":"2023-10-11T11:15:31.671363+0000","last_unstale":"2023-10-11T11:19:00.628569+0000","last_undegraded":"2023-10-11T11:19:00.628569+0000","last_fullsized":"2023-10-11T11:15:31.652001+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:01:53.884964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":512,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-10-11T11:19:03.429640+0000","last_change":"2023-10-11T10:43:41.772781+0000","last_active":"2023-10-11T11:19:03.429640+0000","last_peered":"2023-10-11T11:19:03.429640+0000","last_clean":"2023-10-11T11:19:03.429640+0000","last_became_active":"2023-10-11T10:43:41.772443+0000","last_became_peered":"2023-10-11T10:43:41.772443+0000","last_unstale":"2023-10-11T11:19:03.429640+0000","last_undegraded":"2023-10-11T11:19:03.429640+0000","last_fullsized":"2023-10-11T11:19:03.429640+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T14:15:30.547284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":512,"reported_epoch":499,"state":"active+clean","last_fresh":"2023-10-11T11:19:00.630424+0000","last_change":"2023-10-11T10:43:41.772615+0000","last_active":"2023-10-11T11:19:00.630424+0000","last_peered":"2023-10-11T11:19:00.630424+0000","last_clean":"2023-10-11T11:19:00.630424+0000","last_became_active":"2023-10-11T10:43:41.772290+0000","last_became_peered":"2023-10-11T10:43:41.772290+0000","last_unstale":"2023-10-11T11:19:00.630424+0000","last_undegraded":"2023-10-11T11:19:00.630424+0000","last_fullsized":"2023-10-11T11:19:00.630424+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:26:06.909963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":647,"reported_epoch":499,"state":"active+undersized+degraded","last_fresh":"2023-10-11T11:19:00.625113+0000","last_change":"2023-10-11T11:15:31.654022+0000","last_active":"2023-10-11T11:19:00.625113+0000","last_peered":"2023-10-11T11:19:00.625113+0000","last_clean":"2023-10-11T11:15:27.636884+0000","last_became_active":"2023-10-11T11:15:31.654022+0000","last_became_peered":"2023-10-11T11:15:31.654022+0000","last_unstale":"2023-10-11T11:19:00.625113+0000","last_undegraded":"2023-10-11T11:15:31.646946+0000","last_fullsized":"2023-10-11T11:15:31.646751+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:36.739637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:36.739637+0000","last_clean_scrub_stamp":"2023-10-11T10:43:36.739637+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T11:33:33.696447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":519,"reported_epoch":499,"state":"active+clean+remapped","last_fresh":"2023-10-11T11:19:01.882635+0000","last_change":"2023-10-11T11:15:33.660200+0000","last_active":"2023-10-11T11:19:01.882635+0000","last_peered":"2023-10-11T11:19:01.882635+0000","last_clean":"2023-10-11T11:19:01.882635+0000","last_became_active":"2023-10-11T11:15:33.659619+0000","last_became_peered":"2023-10-11T11:15:33.659619+0000","last_unstale":"2023-10-11T11:19:01.882635+0000","last_undegraded":"2023-10-11T11:19:01.882635+0000","last_fullsized":"2023-10-11T11:19:01.882635+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T17:21:16.866392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":516,"reported_epoch":499,"state":"active+undersized","last_fresh":"2023-10-11T11:19:00.633628+0000","last_change":"2023-10-11T11:15:31.670813+0000","last_active":"2023-10-11T11:19:00.633628+0000","last_peered":"2023-10-11T11:19:00.633628+0000","last_clean":"2023-10-11T11:15:27.638219+0000","last_became_active":"2023-10-11T11:15:31.670813+0000","last_became_peered":"2023-10-11T11:15:31.670813+0000","last_unstale":"2023-10-11T11:19:00.633628+0000","last_undegraded":"2023-10-11T11:19:00.633628+0000","last_fullsized":"2023-10-11T11:15:31.658785+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:41:50.872346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":516,"reported_epoch":499,"state":"active+undersized","last_fresh":"2023-10-11T11:19:00.631668+0000","last_change":"2023-10-11T11:15:31.666378+0000","last_active":"2023-10-11T11:19:00.631668+0000","last_peered":"2023-10-11T11:19:00.631668+0000","last_clean":"2023-10-11T11:15:27.638303+0000","last_became_active":"2023-10-11T11:15:31.666378+0000","last_became_peered":"2023-10-11T11:15:31.666378+0000","last_unstale":"2023-10-11T11:19:00.631668+0000","last_undegraded":"2023-10-11T11:19:00.631668+0000","last_fullsized":"2023-10-11T11:15:31.649186+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T15:35:19.331209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":516,"reported_epoch":499,"state":"active+undersized","last_fresh":"2023-10-11T11:19:00.636960+0000","last_change":"2023-10-11T11:15:31.679465+0000","last_active":"2023-10-11T11:19:00.636960+0000","last_peered":"2023-10-11T11:19:00.636960+0000","last_clean":"2023-10-11T11:15:27.638749+0000","last_became_active":"2023-10-11T11:15:31.679465+0000","last_became_peered":"2023-10-11T11:15:31.679465+0000","last_unstale":"2023-10-11T11:19:00.636960+0000","last_undegraded":"2023-10-11T11:19:00.636960+0000","last_fullsized":"2023-10-11T11:15:31.659135+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T16:03:42.656692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":11,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738797,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":785348,"kb_used_data":5812,"kb_used_omap":96,"kb_used_meta":779423,"kb_avail":93586492,"statfs":{"total":96636764160,"available":95832567808,"internally_reserved":0,"allocated":5951488,"data_stored":3106918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":98609,"internal_metadata":798129871},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 11:14:37 2023","interfaces":[{"interface":"back","average":{"1min":0.314,"5min":0.316,"15min":0.443},"min":{"1min":0.24399999999999999,"5min":0.14799999999999999,"15min":0.14599999999999999},"max":{"1min":0.48499999999999999,"5min":1.712,"15min":9.6400000000000006},"last":0.31},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.35499999999999998,"15min":0.41299999999999998},"min":{"1min":0.20899999999999999,"5min":0.17199999999999999,"15min":0.16400000000000001},"max":{"1min":3.8180000000000001,"5min":3.8180000000000001,"15min":6.3200000000000003},"last":0.26500000000000001}]},{"osd":1,"last update":"Wed Oct 11 11:18:39 2023","interfaces":[{"interface":"back","average":{"1min":0.27500000000000002,"5min":0.29299999999999998,"15min":0.32400000000000001},"min":{"1min":0.154,"5min":0.154,"15min":0.154},"max":{"1min":0.41999999999999998,"5min":0.59299999999999997,"15min":2.9540000000000002},"last":0.17699999999999999},{"interface":"front","average":{"1min":0.307,"5min":0.35899999999999999,"15min":0.36499999999999999},"min":{"1min":0.18099999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.45100000000000001,"5min":3.2719999999999998,"15min":3.2719999999999998},"last":0.20899999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738756,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1484496,"kb_used_data":406792,"kb_used_omap":107,"kb_used_meta":1077588,"kb_avail":92887344,"statfs":{"total":96636764160,"available":95116640256,"internally_reserved":0,"allocated":416555008,"data_stored":413648921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110309,"internal_metadata":1103450395},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Oct 11 11:14:30 2023","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.35399999999999998,"15min":0.40600000000000003},"min":{"1min":0.18099999999999999,"5min":0.155,"15min":0.14899999999999999},"max":{"1min":1.4319999999999999,"5min":2.3460000000000001,"15min":6.593},"last":0.26200000000000001},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.39100000000000001,"15min":0.48199999999999998},"min":{"1min":0.189,"5min":0.189,"15min":0.183},"max":{"1min":2.2120000000000002,"5min":2.4089999999999998,"15min":6.5960000000000001},"last":0.316}]},{"osd":2,"last update":"Wed Oct 11 11:14:30 2023","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.35499999999999998,"15min":0.47099999999999997},"min":{"1min":0.20300000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":1.2490000000000001,"5min":2.665,"15min":8.4730000000000008},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.375,"15min":0.442},"min":{"1min":0.23200000000000001,"5min":0.20499999999999999,"15min":0.19600000000000001},"max":{"1min":1.28,"5min":2.6230000000000002,"15min":7.0129999999999999},"last":0.33600000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738798,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":826440,"kb_used_data":6392,"kb_used_omap":88,"kb_used_meta":819943,"kb_avail":93545400,"statfs":{"total":96636764160,"available":95790489600,"internally_reserved":0,"allocated":6545408,"data_stored":3697286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":90523,"internal_metadata":839622245},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 11:14:33 2023","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.42799999999999999,"15min":0.41099999999999998},"min":{"1min":0.151,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":1.2809999999999999,"5min":3.5739999999999998,"15min":8.2479999999999993},"last":0.223},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.48299999999999998,"15min":0.49199999999999999},"min":{"1min":0.20699999999999999,"5min":0.182,"15min":0.17999999999999999},"max":{"1min":1.323,"5min":3.6080000000000001,"15min":8.2870000000000008},"last":0.28399999999999997}]},{"osd":2,"last update":"Wed Oct 11 11:18:40 2023","interfaces":[{"interface":"back","average":{"1min":0.33400000000000002,"5min":0.378,"15min":0.35599999999999998},"min":{"1min":0.16500000000000001,"5min":0.14699999999999999,"15min":0.14699999999999999},"max":{"1min":1.244,"5min":7.4829999999999997,"15min":10.617000000000001},"last":0.23400000000000001},{"interface":"front","average":{"1min":0.34599999999999997,"5min":0.39000000000000001,"15min":0.34999999999999998},"min":{"1min":0.20399999999999999,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.81100000000000005,"5min":7.5350000000000001,"15min":7.5350000000000001},"last":0.28799999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-11T11:19:08.990 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-10-11T11:19:09.307 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T11:19:09.309 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2023-10-11T11:19:09.320 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":1565,"stamp":"2023-10-11T11:19:08.645514+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":11,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":148,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3096284,"kb_used_data":418996,"kb_used_omap":292,"kb_used_meta":2676955,"kb_avail":280019236,"statfs":{"total":289910292480,"available":286739697664,"internally_reserved":0,"allocated":429051904,"data_stored":420453125,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299441,"internal_metadata":2741202511},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19,"num_objects":-8,"num_object_clones":0,"num_object_copies":-16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-8,"num_whiteouts":0,"num_read":-2040,"num_read_kb":-1943,"num_write":-383,"num_write_kb":-303,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.023021"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":516,"reported_epoch":499,"state":"active+undersized","last_fresh":"2023-10-11T11:19:00.633571+0000","last_change":"2023-10-11T11:15:31.661747+0000","last_active":"2023-10-11T11:19:00.633571+0000","last_peered":"2023-10-11T11:19:00.633571+0000","last_clean":"2023-10-11T11:15:27.638215+0000","last_became_active":"2023-10-11T11:15:31.661747+0000","last_became_peered":"2023-10-11T11:15:31.661747+0000","last_unstale":"2023-10-11T11:19:00.633571+0000","last_undegraded":"2023-10-11T11:19:00.633571+0000","last_fullsized":"2023-10-11T11:15:31.649336+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:15:25.047039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":28,"reported_epoch":499,"state":"active+undersized","last_fresh":"2023-10-11T11:19:00.628569+0000","last_change":"2023-10-11T11:15:31.671363+0000","last_active":"2023-10-11T11:19:00.628569+0000","last_peered":"2023-10-11T11:19:00.628569+0000","last_clean":"2023-10-11T10:43:40.751300+0000","last_became_active":"2023-10-11T11:15:31.671363+0000","last_became_peered":"2023-10-11T11:15:31.671363+0000","last_unstale":"2023-10-11T11:19:00.628569+0000","last_undegraded":"2023-10-11T11:19:00.628569+0000","last_fullsized":"2023-10-11T11:15:31.652001+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:01:53.884964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":512,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-10-11T11:19:03.429640+0000","last_change":"2023-10-11T10:43:41.772781+0000","last_active":"2023-10-11T11:19:03.429640+0000","last_peered":"2023-10-11T11:19:03.429640+0000","last_clean":"2023-10-11T11:19:03.429640+0000","last_became_active":"2023-10-11T10:43:41.772443+0000","last_became_peered":"2023-10-11T10:43:41.772443+0000","last_unstale":"2023-10-11T11:19:03.429640+0000","last_undegraded":"2023-10-11T11:19:03.429640+0000","last_fullsized":"2023-10-11T11:19:03.429640+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T14:15:30.547284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":512,"reported_epoch":499,"state":"active+clean","last_fresh":"2023-10-11T11:19:00.630424+0000","last_change":"2023-10-11T10:43:41.772615+0000","last_active":"2023-10-11T11:19:00.630424+0000","last_peered":"2023-10-11T11:19:00.630424+0000","last_clean":"2023-10-11T11:19:00.630424+0000","last_became_active":"2023-10-11T10:43:41.772290+0000","last_became_peered":"2023-10-11T10:43:41.772290+0000","last_unstale":"2023-10-11T11:19:00.630424+0000","last_undegraded":"2023-10-11T11:19:00.630424+0000","last_fullsized":"2023-10-11T11:19:00.630424+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:26:06.909963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":647,"reported_epoch":499,"state":"active+undersized+degraded","last_fresh":"2023-10-11T11:19:00.625113+0000","last_change":"2023-10-11T11:15:31.654022+0000","last_active":"2023-10-11T11:19:00.625113+0000","last_peered":"2023-10-11T11:19:00.625113+0000","last_clean":"2023-10-11T11:15:27.636884+0000","last_became_active":"2023-10-11T11:15:31.654022+0000","last_became_peered":"2023-10-11T11:15:31.654022+0000","last_unstale":"2023-10-11T11:19:00.625113+0000","last_undegraded":"2023-10-11T11:15:31.646946+0000","last_fullsized":"2023-10-11T11:15:31.646751+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:36.739637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:36.739637+0000","last_clean_scrub_stamp":"2023-10-11T10:43:36.739637+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T11:33:33.696447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":519,"reported_epoch":499,"state":"active+clean+remapped","last_fresh":"2023-10-11T11:19:01.882635+0000","last_change":"2023-10-11T11:15:33.660200+0000","last_active":"2023-10-11T11:19:01.882635+0000","last_peered":"2023-10-11T11:19:01.882635+0000","last_clean":"2023-10-11T11:19:01.882635+0000","last_became_active":"2023-10-11T11:15:33.659619+0000","last_became_peered":"2023-10-11T11:15:33.659619+0000","last_unstale":"2023-10-11T11:19:01.882635+0000","last_undegraded":"2023-10-11T11:19:01.882635+0000","last_fullsized":"2023-10-11T11:19:01.882635+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T17:21:16.866392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":516,"reported_epoch":499,"state":"active+undersized","last_fresh":"2023-10-11T11:19:00.633628+0000","last_change":"2023-10-11T11:15:31.670813+0000","last_active":"2023-10-11T11:19:00.633628+0000","last_peered":"2023-10-11T11:19:00.633628+0000","last_clean":"2023-10-11T11:15:27.638219+0000","last_became_active":"2023-10-11T11:15:31.670813+0000","last_became_peered":"2023-10-11T11:15:31.670813+0000","last_unstale":"2023-10-11T11:19:00.633628+0000","last_undegraded":"2023-10-11T11:19:00.633628+0000","last_fullsized":"2023-10-11T11:15:31.658785+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:41:50.872346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":516,"reported_epoch":499,"state":"active+undersized","last_fresh":"2023-10-11T11:19:00.631668+0000","last_change":"2023-10-11T11:15:31.666378+0000","last_active":"2023-10-11T11:19:00.631668+0000","last_peered":"2023-10-11T11:19:00.631668+0000","last_clean":"2023-10-11T11:15:27.638303+0000","last_became_active":"2023-10-11T11:15:31.666378+0000","last_became_peered":"2023-10-11T11:15:31.666378+0000","last_unstale":"2023-10-11T11:19:00.631668+0000","last_undegraded":"2023-10-11T11:19:00.631668+0000","last_fullsized":"2023-10-11T11:15:31.649186+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T15:35:19.331209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":516,"reported_epoch":499,"state":"active+undersized","last_fresh":"2023-10-11T11:19:00.636960+0000","last_change":"2023-10-11T11:15:31.679465+0000","last_active":"2023-10-11T11:19:00.636960+0000","last_peered":"2023-10-11T11:19:00.636960+0000","last_clean":"2023-10-11T11:15:27.638749+0000","last_became_active":"2023-10-11T11:15:31.679465+0000","last_became_peered":"2023-10-11T11:15:31.679465+0000","last_unstale":"2023-10-11T11:19:00.636960+0000","last_undegraded":"2023-10-11T11:19:00.636960+0000","last_fullsized":"2023-10-11T11:15:31.659135+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T16:03:42.656692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":11,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738797,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":785348,"kb_used_data":5812,"kb_used_omap":96,"kb_used_meta":779423,"kb_avail":93586492,"statfs":{"total":96636764160,"available":95832567808,"internally_reserved":0,"allocated":5951488,"data_stored":3106918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":98609,"internal_metadata":798129871},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 11:14:37 2023","interfaces":[{"interface":"back","average":{"1min":0.314,"5min":0.316,"15min":0.443},"min":{"1min":0.24399999999999999,"5min":0.14799999999999999,"15min":0.14599999999999999},"max":{"1min":0.48499999999999999,"5min":1.712,"15min":9.6400000000000006},"last":0.31},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.35499999999999998,"15min":0.41299999999999998},"min":{"1min":0.20899999999999999,"5min":0.17199999999999999,"15min":0.16400000000000001},"max":{"1min":3.8180000000000001,"5min":3.8180000000000001,"15min":6.3200000000000003},"last":0.26500000000000001}]},{"osd":1,"last update":"Wed Oct 11 11:18:39 2023","interfaces":[{"interface":"back","average":{"1min":0.27500000000000002,"5min":0.29299999999999998,"15min":0.32400000000000001},"min":{"1min":0.154,"5min":0.154,"15min":0.154},"max":{"1min":0.41999999999999998,"5min":0.59299999999999997,"15min":2.9540000000000002},"last":0.17699999999999999},{"interface":"front","average":{"1min":0.307,"5min":0.35899999999999999,"15min":0.36499999999999999},"min":{"1min":0.18099999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.45100000000000001,"5min":3.2719999999999998,"15min":3.2719999999999998},"last":0.20899999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738756,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1484496,"kb_used_data":406792,"kb_used_omap":107,"kb_used_meta":1077588,"kb_avail":92887344,"statfs":{"total":96636764160,"available":95116640256,"internally_reserved":0,"allocated":416555008,"data_stored":413648921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110309,"internal_metadata":1103450395},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Oct 11 11:14:30 2023","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.35399999999999998,"15min":0.40600000000000003},"min":{"1min":0.18099999999999999,"5min":0.155,"15min":0.14899999999999999},"max":{"1min":1.4319999999999999,"5min":2.3460000000000001,"15min":6.593},"last":0.26200000000000001},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.39100000000000001,"15min":0.48199999999999998},"min":{"1min":0.189,"5min":0.189,"15min":0.183},"max":{"1min":2.2120000000000002,"5min":2.4089999999999998,"15min":6.5960000000000001},"last":0.316}]},{"osd":2,"last update":"Wed Oct 11 11:14:30 2023","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.35499999999999998,"15min":0.47099999999999997},"min":{"1min":0.20300000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":1.2490000000000001,"5min":2.665,"15min":8.4730000000000008},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.375,"15min":0.442},"min":{"1min":0.23200000000000001,"5min":0.20499999999999999,"15min":0.19600000000000001},"max":{"1min":1.28,"5min":2.6230000000000002,"15min":7.0129999999999999},"last":0.33600000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738798,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":826440,"kb_used_data":6392,"kb_used_omap":88,"kb_used_meta":819943,"kb_avail":93545400,"statfs":{"total":96636764160,"available":95790489600,"internally_reserved":0,"allocated":6545408,"data_stored":3697286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":90523,"internal_metadata":839622245},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 11:14:33 2023","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.42799999999999999,"15min":0.41099999999999998},"min":{"1min":0.151,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":1.2809999999999999,"5min":3.5739999999999998,"15min":8.2479999999999993},"last":0.223},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.48299999999999998,"15min":0.49199999999999999},"min":{"1min":0.20699999999999999,"5min":0.182,"15min":0.17999999999999999},"max":{"1min":1.323,"5min":3.6080000000000001,"15min":8.2870000000000008},"last":0.28399999999999997}]},{"osd":2,"last update":"Wed Oct 11 11:18:40 2023","interfaces":[{"interface":"back","average":{"1min":0.33400000000000002,"5min":0.378,"15min":0.35599999999999998},"min":{"1min":0.16500000000000001,"5min":0.14699999999999999,"15min":0.14699999999999999},"max":{"1min":1.244,"5min":7.4829999999999997,"15min":10.617000000000001},"last":0.23400000000000001},{"interface":"front","average":{"1min":0.34599999999999997,"5min":0.39000000000000001,"15min":0.34999999999999998},"min":{"1min":0.20399999999999999,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.81100000000000005,"5min":7.5350000000000001,"15min":7.5350000000000001},"last":0.28799999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-11T11:19:09.321 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-10-11T11:19:09.322 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 516, 'reported_epoch': 499, 'state': 'active+undersized', 'last_fresh': '2023-10-11T11:19:00.633571+0000', 'last_change': '2023-10-11T11:15:31.661747+0000', 'last_active': '2023-10-11T11:19:00.633571+0000', 'last_peered': '2023-10-11T11:19:00.633571+0000', 'last_clean': '2023-10-11T11:15:27.638215+0000', 'last_became_active': '2023-10-11T11:15:31.661747+0000', 'last_became_peered': '2023-10-11T11:15:31.661747+0000', 'last_unstale': '2023-10-11T11:19:00.633571+0000', 'last_undegraded': '2023-10-11T11:19:00.633571+0000', 'last_fullsized': '2023-10-11T11:15:31.649336+0000', 'mapping_epoch': 480, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_clean_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-12T22:15:25.047039+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-10-11T11:19:09.322 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-10-11T11:19:09.322 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 28, 'reported_epoch': 499, 'state': 'active+undersized', 'last_fresh': '2023-10-11T11:19:00.628569+0000', 'last_change': '2023-10-11T11:15:31.671363+0000', 'last_active': '2023-10-11T11:19:00.628569+0000', 'last_peered': '2023-10-11T11:19:00.628569+0000', 'last_clean': '2023-10-11T10:43:40.751300+0000', 'last_became_active': '2023-10-11T11:15:31.671363+0000', 'last_became_peered': '2023-10-11T11:15:31.671363+0000', 'last_unstale': '2023-10-11T11:19:00.628569+0000', 'last_undegraded': '2023-10-11T11:19:00.628569+0000', 'last_fullsized': '2023-10-11T11:15:31.652001+0000', 'mapping_epoch': 480, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_clean_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-12T21:01:53.884964+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-10-11T11:19:09.323 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-10-11T11:19:09.323 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "11'97", 'reported_seq': 647, 'reported_epoch': 499, 'state': 'active+undersized+degraded', 'last_fresh': '2023-10-11T11:19:00.625113+0000', 'last_change': '2023-10-11T11:15:31.654022+0000', 'last_active': '2023-10-11T11:19:00.625113+0000', 'last_peered': '2023-10-11T11:19:00.625113+0000', 'last_clean': '2023-10-11T11:15:27.636884+0000', 'last_became_active': '2023-10-11T11:15:31.654022+0000', 'last_became_peered': '2023-10-11T11:15:31.654022+0000', 'last_unstale': '2023-10-11T11:19:00.625113+0000', 'last_undegraded': '2023-10-11T11:15:31.646946+0000', 'last_fullsized': '2023-10-11T11:15:31.646751+0000', 'mapping_epoch': 480, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-11T10:43:36.739637+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-11T10:43:36.739637+0000', 'last_clean_scrub_stamp': '2023-10-11T10:43:36.739637+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-12T11:33:33.696447+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-10-11T11:19:09.323 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.0 is not active+clean 2023-10-11T11:19:09.324 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 519, 'reported_epoch': 499, 'state': 'active+clean+remapped', 'last_fresh': '2023-10-11T11:19:01.882635+0000', 'last_change': '2023-10-11T11:15:33.660200+0000', 'last_active': '2023-10-11T11:19:01.882635+0000', 'last_peered': '2023-10-11T11:19:01.882635+0000', 'last_clean': '2023-10-11T11:19:01.882635+0000', 'last_became_active': '2023-10-11T11:15:33.659619+0000', 'last_became_peered': '2023-10-11T11:15:33.659619+0000', 'last_unstale': '2023-10-11T11:19:01.882635+0000', 'last_undegraded': '2023-10-11T11:19:01.882635+0000', 'last_fullsized': '2023-10-11T11:19:01.882635+0000', 'mapping_epoch': 482, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 483, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_clean_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-12T17:21:16.866392+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2, 1], 'avail_no_missing': ['2', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-10-11T11:19:09.324 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-10-11T11:19:09.324 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 516, 'reported_epoch': 499, 'state': 'active+undersized', 'last_fresh': '2023-10-11T11:19:00.633628+0000', 'last_change': '2023-10-11T11:15:31.670813+0000', 'last_active': '2023-10-11T11:19:00.633628+0000', 'last_peered': '2023-10-11T11:19:00.633628+0000', 'last_clean': '2023-10-11T11:15:27.638219+0000', 'last_became_active': '2023-10-11T11:15:31.670813+0000', 'last_became_peered': '2023-10-11T11:15:31.670813+0000', 'last_unstale': '2023-10-11T11:19:00.633628+0000', 'last_undegraded': '2023-10-11T11:19:00.633628+0000', 'last_fullsized': '2023-10-11T11:15:31.658785+0000', 'mapping_epoch': 480, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_clean_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-12T21:41:50.872346+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-10-11T11:19:09.324 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-10-11T11:19:09.325 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 516, 'reported_epoch': 499, 'state': 'active+undersized', 'last_fresh': '2023-10-11T11:19:00.631668+0000', 'last_change': '2023-10-11T11:15:31.666378+0000', 'last_active': '2023-10-11T11:19:00.631668+0000', 'last_peered': '2023-10-11T11:19:00.631668+0000', 'last_clean': '2023-10-11T11:15:27.638303+0000', 'last_became_active': '2023-10-11T11:15:31.666378+0000', 'last_became_peered': '2023-10-11T11:15:31.666378+0000', 'last_unstale': '2023-10-11T11:19:00.631668+0000', 'last_undegraded': '2023-10-11T11:19:00.631668+0000', 'last_fullsized': '2023-10-11T11:15:31.649186+0000', 'mapping_epoch': 480, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_clean_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-12T15:35:19.331209+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-10-11T11:19:09.325 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-10-11T11:19:09.325 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 516, 'reported_epoch': 499, 'state': 'active+undersized', 'last_fresh': '2023-10-11T11:19:00.636960+0000', 'last_change': '2023-10-11T11:15:31.679465+0000', 'last_active': '2023-10-11T11:19:00.636960+0000', 'last_peered': '2023-10-11T11:19:00.636960+0000', 'last_clean': '2023-10-11T11:15:27.638749+0000', 'last_became_active': '2023-10-11T11:15:31.679465+0000', 'last_became_peered': '2023-10-11T11:15:31.679465+0000', 'last_unstale': '2023-10-11T11:19:00.636960+0000', 'last_undegraded': '2023-10-11T11:19:00.636960+0000', 'last_fullsized': '2023-10-11T11:15:31.659135+0000', 'mapping_epoch': 480, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_clean_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-12T16:03:42.656692+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-10-11T11:19:09.326 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-10-11T11:19:09.717 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T11:19:09.718 INFO:teuthology.orchestra.run.smithi111.stdout:{"fsid":"022ce6f2-9a5b-45ad-8b77-a1332d92f228","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2144,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":500,"num_osds":3,"num_up_osds":2,"osd_up_since":1697022930,"num_in_osds":3,"osd_in_since":1697021005,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":2},{"state_name":"active+undersized+degraded","count":1},{"state_name":"active+clean+remapped","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3170594816,"bytes_avail":286739697664,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":36,"modified":"2023-10-11T11:19:04.644408+0000","services":{}},"progress_events":{}} 2023-10-11T11:19:09.731 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-10-11T11:19:09.732 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-10-11T11:19:10.047 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T11:19:10.049 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2023-10-11T11:19:10.061 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":1565,"stamp":"2023-10-11T11:19:08.645514+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":11,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":148,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3096284,"kb_used_data":418996,"kb_used_omap":292,"kb_used_meta":2676955,"kb_avail":280019236,"statfs":{"total":289910292480,"available":286739697664,"internally_reserved":0,"allocated":429051904,"data_stored":420453125,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299441,"internal_metadata":2741202511},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19,"num_objects":-8,"num_object_clones":0,"num_object_copies":-16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-8,"num_whiteouts":0,"num_read":-2040,"num_read_kb":-1943,"num_write":-383,"num_write_kb":-303,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.023021"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":516,"reported_epoch":499,"state":"active+undersized","last_fresh":"2023-10-11T11:19:00.633571+0000","last_change":"2023-10-11T11:15:31.661747+0000","last_active":"2023-10-11T11:19:00.633571+0000","last_peered":"2023-10-11T11:19:00.633571+0000","last_clean":"2023-10-11T11:15:27.638215+0000","last_became_active":"2023-10-11T11:15:31.661747+0000","last_became_peered":"2023-10-11T11:15:31.661747+0000","last_unstale":"2023-10-11T11:19:00.633571+0000","last_undegraded":"2023-10-11T11:19:00.633571+0000","last_fullsized":"2023-10-11T11:15:31.649336+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:15:25.047039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":28,"reported_epoch":499,"state":"active+undersized","last_fresh":"2023-10-11T11:19:00.628569+0000","last_change":"2023-10-11T11:15:31.671363+0000","last_active":"2023-10-11T11:19:00.628569+0000","last_peered":"2023-10-11T11:19:00.628569+0000","last_clean":"2023-10-11T10:43:40.751300+0000","last_became_active":"2023-10-11T11:15:31.671363+0000","last_became_peered":"2023-10-11T11:15:31.671363+0000","last_unstale":"2023-10-11T11:19:00.628569+0000","last_undegraded":"2023-10-11T11:19:00.628569+0000","last_fullsized":"2023-10-11T11:15:31.652001+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:01:53.884964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":512,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-10-11T11:19:03.429640+0000","last_change":"2023-10-11T10:43:41.772781+0000","last_active":"2023-10-11T11:19:03.429640+0000","last_peered":"2023-10-11T11:19:03.429640+0000","last_clean":"2023-10-11T11:19:03.429640+0000","last_became_active":"2023-10-11T10:43:41.772443+0000","last_became_peered":"2023-10-11T10:43:41.772443+0000","last_unstale":"2023-10-11T11:19:03.429640+0000","last_undegraded":"2023-10-11T11:19:03.429640+0000","last_fullsized":"2023-10-11T11:19:03.429640+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T14:15:30.547284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":512,"reported_epoch":499,"state":"active+clean","last_fresh":"2023-10-11T11:19:00.630424+0000","last_change":"2023-10-11T10:43:41.772615+0000","last_active":"2023-10-11T11:19:00.630424+0000","last_peered":"2023-10-11T11:19:00.630424+0000","last_clean":"2023-10-11T11:19:00.630424+0000","last_became_active":"2023-10-11T10:43:41.772290+0000","last_became_peered":"2023-10-11T10:43:41.772290+0000","last_unstale":"2023-10-11T11:19:00.630424+0000","last_undegraded":"2023-10-11T11:19:00.630424+0000","last_fullsized":"2023-10-11T11:19:00.630424+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:26:06.909963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":647,"reported_epoch":499,"state":"active+undersized+degraded","last_fresh":"2023-10-11T11:19:00.625113+0000","last_change":"2023-10-11T11:15:31.654022+0000","last_active":"2023-10-11T11:19:00.625113+0000","last_peered":"2023-10-11T11:19:00.625113+0000","last_clean":"2023-10-11T11:15:27.636884+0000","last_became_active":"2023-10-11T11:15:31.654022+0000","last_became_peered":"2023-10-11T11:15:31.654022+0000","last_unstale":"2023-10-11T11:19:00.625113+0000","last_undegraded":"2023-10-11T11:15:31.646946+0000","last_fullsized":"2023-10-11T11:15:31.646751+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:36.739637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:36.739637+0000","last_clean_scrub_stamp":"2023-10-11T10:43:36.739637+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T11:33:33.696447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":519,"reported_epoch":499,"state":"active+clean+remapped","last_fresh":"2023-10-11T11:19:01.882635+0000","last_change":"2023-10-11T11:15:33.660200+0000","last_active":"2023-10-11T11:19:01.882635+0000","last_peered":"2023-10-11T11:19:01.882635+0000","last_clean":"2023-10-11T11:19:01.882635+0000","last_became_active":"2023-10-11T11:15:33.659619+0000","last_became_peered":"2023-10-11T11:15:33.659619+0000","last_unstale":"2023-10-11T11:19:01.882635+0000","last_undegraded":"2023-10-11T11:19:01.882635+0000","last_fullsized":"2023-10-11T11:19:01.882635+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T17:21:16.866392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":516,"reported_epoch":499,"state":"active+undersized","last_fresh":"2023-10-11T11:19:00.633628+0000","last_change":"2023-10-11T11:15:31.670813+0000","last_active":"2023-10-11T11:19:00.633628+0000","last_peered":"2023-10-11T11:19:00.633628+0000","last_clean":"2023-10-11T11:15:27.638219+0000","last_became_active":"2023-10-11T11:15:31.670813+0000","last_became_peered":"2023-10-11T11:15:31.670813+0000","last_unstale":"2023-10-11T11:19:00.633628+0000","last_undegraded":"2023-10-11T11:19:00.633628+0000","last_fullsized":"2023-10-11T11:15:31.658785+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:41:50.872346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":516,"reported_epoch":499,"state":"active+undersized","last_fresh":"2023-10-11T11:19:00.631668+0000","last_change":"2023-10-11T11:15:31.666378+0000","last_active":"2023-10-11T11:19:00.631668+0000","last_peered":"2023-10-11T11:19:00.631668+0000","last_clean":"2023-10-11T11:15:27.638303+0000","last_became_active":"2023-10-11T11:15:31.666378+0000","last_became_peered":"2023-10-11T11:15:31.666378+0000","last_unstale":"2023-10-11T11:19:00.631668+0000","last_undegraded":"2023-10-11T11:19:00.631668+0000","last_fullsized":"2023-10-11T11:15:31.649186+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T15:35:19.331209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":516,"reported_epoch":499,"state":"active+undersized","last_fresh":"2023-10-11T11:19:00.636960+0000","last_change":"2023-10-11T11:15:31.679465+0000","last_active":"2023-10-11T11:19:00.636960+0000","last_peered":"2023-10-11T11:19:00.636960+0000","last_clean":"2023-10-11T11:15:27.638749+0000","last_became_active":"2023-10-11T11:15:31.679465+0000","last_became_peered":"2023-10-11T11:15:31.679465+0000","last_unstale":"2023-10-11T11:19:00.636960+0000","last_undegraded":"2023-10-11T11:19:00.636960+0000","last_fullsized":"2023-10-11T11:15:31.659135+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T16:03:42.656692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":11,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738797,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":785348,"kb_used_data":5812,"kb_used_omap":96,"kb_used_meta":779423,"kb_avail":93586492,"statfs":{"total":96636764160,"available":95832567808,"internally_reserved":0,"allocated":5951488,"data_stored":3106918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":98609,"internal_metadata":798129871},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 11:14:37 2023","interfaces":[{"interface":"back","average":{"1min":0.314,"5min":0.316,"15min":0.443},"min":{"1min":0.24399999999999999,"5min":0.14799999999999999,"15min":0.14599999999999999},"max":{"1min":0.48499999999999999,"5min":1.712,"15min":9.6400000000000006},"last":0.31},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.35499999999999998,"15min":0.41299999999999998},"min":{"1min":0.20899999999999999,"5min":0.17199999999999999,"15min":0.16400000000000001},"max":{"1min":3.8180000000000001,"5min":3.8180000000000001,"15min":6.3200000000000003},"last":0.26500000000000001}]},{"osd":1,"last update":"Wed Oct 11 11:18:39 2023","interfaces":[{"interface":"back","average":{"1min":0.27500000000000002,"5min":0.29299999999999998,"15min":0.32400000000000001},"min":{"1min":0.154,"5min":0.154,"15min":0.154},"max":{"1min":0.41999999999999998,"5min":0.59299999999999997,"15min":2.9540000000000002},"last":0.17699999999999999},{"interface":"front","average":{"1min":0.307,"5min":0.35899999999999999,"15min":0.36499999999999999},"min":{"1min":0.18099999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.45100000000000001,"5min":3.2719999999999998,"15min":3.2719999999999998},"last":0.20899999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738756,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1484496,"kb_used_data":406792,"kb_used_omap":107,"kb_used_meta":1077588,"kb_avail":92887344,"statfs":{"total":96636764160,"available":95116640256,"internally_reserved":0,"allocated":416555008,"data_stored":413648921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110309,"internal_metadata":1103450395},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Oct 11 11:14:30 2023","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.35399999999999998,"15min":0.40600000000000003},"min":{"1min":0.18099999999999999,"5min":0.155,"15min":0.14899999999999999},"max":{"1min":1.4319999999999999,"5min":2.3460000000000001,"15min":6.593},"last":0.26200000000000001},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.39100000000000001,"15min":0.48199999999999998},"min":{"1min":0.189,"5min":0.189,"15min":0.183},"max":{"1min":2.2120000000000002,"5min":2.4089999999999998,"15min":6.5960000000000001},"last":0.316}]},{"osd":2,"last update":"Wed Oct 11 11:14:30 2023","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.35499999999999998,"15min":0.47099999999999997},"min":{"1min":0.20300000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":1.2490000000000001,"5min":2.665,"15min":8.4730000000000008},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.375,"15min":0.442},"min":{"1min":0.23200000000000001,"5min":0.20499999999999999,"15min":0.19600000000000001},"max":{"1min":1.28,"5min":2.6230000000000002,"15min":7.0129999999999999},"last":0.33600000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738798,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":826440,"kb_used_data":6392,"kb_used_omap":88,"kb_used_meta":819943,"kb_avail":93545400,"statfs":{"total":96636764160,"available":95790489600,"internally_reserved":0,"allocated":6545408,"data_stored":3697286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":90523,"internal_metadata":839622245},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 11:14:33 2023","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.42799999999999999,"15min":0.41099999999999998},"min":{"1min":0.151,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":1.2809999999999999,"5min":3.5739999999999998,"15min":8.2479999999999993},"last":0.223},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.48299999999999998,"15min":0.49199999999999999},"min":{"1min":0.20699999999999999,"5min":0.182,"15min":0.17999999999999999},"max":{"1min":1.323,"5min":3.6080000000000001,"15min":8.2870000000000008},"last":0.28399999999999997}]},{"osd":2,"last update":"Wed Oct 11 11:18:40 2023","interfaces":[{"interface":"back","average":{"1min":0.33400000000000002,"5min":0.378,"15min":0.35599999999999998},"min":{"1min":0.16500000000000001,"5min":0.14699999999999999,"15min":0.14699999999999999},"max":{"1min":1.244,"5min":7.4829999999999997,"15min":10.617000000000001},"last":0.23400000000000001},{"interface":"front","average":{"1min":0.34599999999999997,"5min":0.39000000000000001,"15min":0.34999999999999998},"min":{"1min":0.20399999999999999,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.81100000000000005,"5min":7.5350000000000001,"15min":7.5350000000000001},"last":0.28799999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-11T11:19:13.062 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-10-11T11:19:13.392 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T11:19:13.394 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2023-10-11T11:19:13.406 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":1567,"stamp":"2023-10-11T11:19:12.646621+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":11,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":107,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3096284,"kb_used_data":418996,"kb_used_omap":292,"kb_used_meta":2676955,"kb_avail":280019236,"statfs":{"total":289910292480,"available":286739697664,"internally_reserved":0,"allocated":429051904,"data_stored":420453125,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299441,"internal_metadata":2741202511},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19,"num_objects":-8,"num_object_clones":0,"num_object_copies":-16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-8,"num_whiteouts":0,"num_read":-2040,"num_read_kb":-1943,"num_write":-383,"num_write_kb":-303,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002922"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":517,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-10-11T11:19:02.772828+0000","last_change":"2023-10-11T11:15:31.661747+0000","last_active":"2023-10-11T11:19:02.772828+0000","last_peered":"2023-10-11T11:19:02.772828+0000","last_clean":"2023-10-11T11:15:27.638215+0000","last_became_active":"2023-10-11T11:15:31.661747+0000","last_became_peered":"2023-10-11T11:15:31.661747+0000","last_unstale":"2023-10-11T11:19:02.772828+0000","last_undegraded":"2023-10-11T11:19:02.772828+0000","last_fullsized":"2023-10-11T11:15:31.649336+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:15:25.047039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":28,"reported_epoch":499,"state":"active+undersized","last_fresh":"2023-10-11T11:19:00.628569+0000","last_change":"2023-10-11T11:15:31.671363+0000","last_active":"2023-10-11T11:19:00.628569+0000","last_peered":"2023-10-11T11:19:00.628569+0000","last_clean":"2023-10-11T10:43:40.751300+0000","last_became_active":"2023-10-11T11:15:31.671363+0000","last_became_peered":"2023-10-11T11:15:31.671363+0000","last_unstale":"2023-10-11T11:19:00.628569+0000","last_undegraded":"2023-10-11T11:19:00.628569+0000","last_fullsized":"2023-10-11T11:15:31.652001+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:01:53.884964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":512,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-10-11T11:19:03.429640+0000","last_change":"2023-10-11T10:43:41.772781+0000","last_active":"2023-10-11T11:19:03.429640+0000","last_peered":"2023-10-11T11:19:03.429640+0000","last_clean":"2023-10-11T11:19:03.429640+0000","last_became_active":"2023-10-11T10:43:41.772443+0000","last_became_peered":"2023-10-11T10:43:41.772443+0000","last_unstale":"2023-10-11T11:19:03.429640+0000","last_undegraded":"2023-10-11T11:19:03.429640+0000","last_fullsized":"2023-10-11T11:19:03.429640+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T14:15:30.547284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":513,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-10-11T11:19:02.772211+0000","last_change":"2023-10-11T10:43:41.772615+0000","last_active":"2023-10-11T11:19:02.772211+0000","last_peered":"2023-10-11T11:19:02.772211+0000","last_clean":"2023-10-11T11:19:02.772211+0000","last_became_active":"2023-10-11T10:43:41.772290+0000","last_became_peered":"2023-10-11T10:43:41.772290+0000","last_unstale":"2023-10-11T11:19:02.772211+0000","last_undegraded":"2023-10-11T11:19:02.772211+0000","last_fullsized":"2023-10-11T11:19:02.772211+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:26:06.909963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":647,"reported_epoch":499,"state":"active+undersized+degraded","last_fresh":"2023-10-11T11:19:00.625113+0000","last_change":"2023-10-11T11:15:31.654022+0000","last_active":"2023-10-11T11:19:00.625113+0000","last_peered":"2023-10-11T11:19:00.625113+0000","last_clean":"2023-10-11T11:15:27.636884+0000","last_became_active":"2023-10-11T11:15:31.654022+0000","last_became_peered":"2023-10-11T11:15:31.654022+0000","last_unstale":"2023-10-11T11:19:00.625113+0000","last_undegraded":"2023-10-11T11:15:31.646946+0000","last_fullsized":"2023-10-11T11:15:31.646751+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:36.739637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:36.739637+0000","last_clean_scrub_stamp":"2023-10-11T10:43:36.739637+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T11:33:33.696447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":519,"reported_epoch":499,"state":"active+clean+remapped","last_fresh":"2023-10-11T11:19:01.882635+0000","last_change":"2023-10-11T11:15:33.660200+0000","last_active":"2023-10-11T11:19:01.882635+0000","last_peered":"2023-10-11T11:19:01.882635+0000","last_clean":"2023-10-11T11:19:01.882635+0000","last_became_active":"2023-10-11T11:15:33.659619+0000","last_became_peered":"2023-10-11T11:15:33.659619+0000","last_unstale":"2023-10-11T11:19:01.882635+0000","last_undegraded":"2023-10-11T11:19:01.882635+0000","last_fullsized":"2023-10-11T11:19:01.882635+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T17:21:16.866392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":516,"reported_epoch":499,"state":"active+undersized","last_fresh":"2023-10-11T11:19:00.633628+0000","last_change":"2023-10-11T11:15:31.670813+0000","last_active":"2023-10-11T11:19:00.633628+0000","last_peered":"2023-10-11T11:19:00.633628+0000","last_clean":"2023-10-11T11:15:27.638219+0000","last_became_active":"2023-10-11T11:15:31.670813+0000","last_became_peered":"2023-10-11T11:15:31.670813+0000","last_unstale":"2023-10-11T11:19:00.633628+0000","last_undegraded":"2023-10-11T11:19:00.633628+0000","last_fullsized":"2023-10-11T11:15:31.658785+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:41:50.872346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":516,"reported_epoch":499,"state":"active+undersized","last_fresh":"2023-10-11T11:19:00.631668+0000","last_change":"2023-10-11T11:15:31.666378+0000","last_active":"2023-10-11T11:19:00.631668+0000","last_peered":"2023-10-11T11:19:00.631668+0000","last_clean":"2023-10-11T11:15:27.638303+0000","last_became_active":"2023-10-11T11:15:31.666378+0000","last_became_peered":"2023-10-11T11:15:31.666378+0000","last_unstale":"2023-10-11T11:19:00.631668+0000","last_undegraded":"2023-10-11T11:19:00.631668+0000","last_fullsized":"2023-10-11T11:15:31.649186+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T15:35:19.331209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":516,"reported_epoch":499,"state":"active+undersized","last_fresh":"2023-10-11T11:19:00.636960+0000","last_change":"2023-10-11T11:15:31.679465+0000","last_active":"2023-10-11T11:19:00.636960+0000","last_peered":"2023-10-11T11:19:00.636960+0000","last_clean":"2023-10-11T11:15:27.638749+0000","last_became_active":"2023-10-11T11:15:31.679465+0000","last_became_peered":"2023-10-11T11:15:31.679465+0000","last_unstale":"2023-10-11T11:19:00.636960+0000","last_undegraded":"2023-10-11T11:19:00.636960+0000","last_fullsized":"2023-10-11T11:15:31.659135+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T16:03:42.656692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":11,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738798,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":785348,"kb_used_data":5812,"kb_used_omap":96,"kb_used_meta":779423,"kb_avail":93586492,"statfs":{"total":96636764160,"available":95832567808,"internally_reserved":0,"allocated":5951488,"data_stored":3106918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":98609,"internal_metadata":798129871},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 11:14:37 2023","interfaces":[{"interface":"back","average":{"1min":0.314,"5min":0.316,"15min":0.443},"min":{"1min":0.24399999999999999,"5min":0.14799999999999999,"15min":0.14599999999999999},"max":{"1min":0.48499999999999999,"5min":1.712,"15min":9.6400000000000006},"last":0.31},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.35499999999999998,"15min":0.41299999999999998},"min":{"1min":0.20899999999999999,"5min":0.17199999999999999,"15min":0.16400000000000001},"max":{"1min":3.8180000000000001,"5min":3.8180000000000001,"15min":6.3200000000000003},"last":0.26500000000000001}]},{"osd":1,"last update":"Wed Oct 11 11:18:39 2023","interfaces":[{"interface":"back","average":{"1min":0.27500000000000002,"5min":0.29299999999999998,"15min":0.32400000000000001},"min":{"1min":0.154,"5min":0.154,"15min":0.154},"max":{"1min":0.41999999999999998,"5min":0.59299999999999997,"15min":2.9540000000000002},"last":0.317},{"interface":"front","average":{"1min":0.307,"5min":0.35899999999999999,"15min":0.36499999999999999},"min":{"1min":0.18099999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.45100000000000001,"5min":3.2719999999999998,"15min":3.2719999999999998},"last":0.39500000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738756,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1484496,"kb_used_data":406792,"kb_used_omap":107,"kb_used_meta":1077588,"kb_avail":92887344,"statfs":{"total":96636764160,"available":95116640256,"internally_reserved":0,"allocated":416555008,"data_stored":413648921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110309,"internal_metadata":1103450395},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Oct 11 11:14:30 2023","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.35399999999999998,"15min":0.40600000000000003},"min":{"1min":0.18099999999999999,"5min":0.155,"15min":0.14899999999999999},"max":{"1min":1.4319999999999999,"5min":2.3460000000000001,"15min":6.593},"last":0.26200000000000001},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.39100000000000001,"15min":0.48199999999999998},"min":{"1min":0.189,"5min":0.189,"15min":0.183},"max":{"1min":2.2120000000000002,"5min":2.4089999999999998,"15min":6.5960000000000001},"last":0.316}]},{"osd":2,"last update":"Wed Oct 11 11:14:30 2023","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.35499999999999998,"15min":0.47099999999999997},"min":{"1min":0.20300000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":1.2490000000000001,"5min":2.665,"15min":8.4730000000000008},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.375,"15min":0.442},"min":{"1min":0.23200000000000001,"5min":0.20499999999999999,"15min":0.19600000000000001},"max":{"1min":1.28,"5min":2.6230000000000002,"15min":7.0129999999999999},"last":0.33600000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738799,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":826440,"kb_used_data":6392,"kb_used_omap":88,"kb_used_meta":819943,"kb_avail":93545400,"statfs":{"total":96636764160,"available":95790489600,"internally_reserved":0,"allocated":6545408,"data_stored":3697286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":90523,"internal_metadata":839622245},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 11:14:33 2023","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.42799999999999999,"15min":0.41099999999999998},"min":{"1min":0.151,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":1.2809999999999999,"5min":3.5739999999999998,"15min":8.2479999999999993},"last":0.223},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.48299999999999998,"15min":0.49199999999999999},"min":{"1min":0.20699999999999999,"5min":0.182,"15min":0.17999999999999999},"max":{"1min":1.323,"5min":3.6080000000000001,"15min":8.2870000000000008},"last":0.28399999999999997}]},{"osd":2,"last update":"Wed Oct 11 11:18:40 2023","interfaces":[{"interface":"back","average":{"1min":0.33400000000000002,"5min":0.378,"15min":0.35599999999999998},"min":{"1min":0.16500000000000001,"5min":0.14699999999999999,"15min":0.14699999999999999},"max":{"1min":1.244,"5min":7.4829999999999997,"15min":10.617000000000001},"last":0.312},{"interface":"front","average":{"1min":0.34599999999999997,"5min":0.39000000000000001,"15min":0.34999999999999998},"min":{"1min":0.20399999999999999,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.81100000000000005,"5min":7.5350000000000001,"15min":7.5350000000000001},"last":0.28100000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-11T11:19:13.407 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-10-11T11:19:13.716 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~222s 2023-10-11T11:19:13.721 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T11:19:13.723 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2023-10-11T11:19:13.735 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":1567,"stamp":"2023-10-11T11:19:12.646621+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":11,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":107,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3096284,"kb_used_data":418996,"kb_used_omap":292,"kb_used_meta":2676955,"kb_avail":280019236,"statfs":{"total":289910292480,"available":286739697664,"internally_reserved":0,"allocated":429051904,"data_stored":420453125,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299441,"internal_metadata":2741202511},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19,"num_objects":-8,"num_object_clones":0,"num_object_copies":-16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-8,"num_whiteouts":0,"num_read":-2040,"num_read_kb":-1943,"num_write":-383,"num_write_kb":-303,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002922"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":517,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-10-11T11:19:02.772828+0000","last_change":"2023-10-11T11:15:31.661747+0000","last_active":"2023-10-11T11:19:02.772828+0000","last_peered":"2023-10-11T11:19:02.772828+0000","last_clean":"2023-10-11T11:15:27.638215+0000","last_became_active":"2023-10-11T11:15:31.661747+0000","last_became_peered":"2023-10-11T11:15:31.661747+0000","last_unstale":"2023-10-11T11:19:02.772828+0000","last_undegraded":"2023-10-11T11:19:02.772828+0000","last_fullsized":"2023-10-11T11:15:31.649336+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:15:25.047039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":28,"reported_epoch":499,"state":"active+undersized","last_fresh":"2023-10-11T11:19:00.628569+0000","last_change":"2023-10-11T11:15:31.671363+0000","last_active":"2023-10-11T11:19:00.628569+0000","last_peered":"2023-10-11T11:19:00.628569+0000","last_clean":"2023-10-11T10:43:40.751300+0000","last_became_active":"2023-10-11T11:15:31.671363+0000","last_became_peered":"2023-10-11T11:15:31.671363+0000","last_unstale":"2023-10-11T11:19:00.628569+0000","last_undegraded":"2023-10-11T11:19:00.628569+0000","last_fullsized":"2023-10-11T11:15:31.652001+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:01:53.884964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":512,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-10-11T11:19:03.429640+0000","last_change":"2023-10-11T10:43:41.772781+0000","last_active":"2023-10-11T11:19:03.429640+0000","last_peered":"2023-10-11T11:19:03.429640+0000","last_clean":"2023-10-11T11:19:03.429640+0000","last_became_active":"2023-10-11T10:43:41.772443+0000","last_became_peered":"2023-10-11T10:43:41.772443+0000","last_unstale":"2023-10-11T11:19:03.429640+0000","last_undegraded":"2023-10-11T11:19:03.429640+0000","last_fullsized":"2023-10-11T11:19:03.429640+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T14:15:30.547284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":513,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-10-11T11:19:02.772211+0000","last_change":"2023-10-11T10:43:41.772615+0000","last_active":"2023-10-11T11:19:02.772211+0000","last_peered":"2023-10-11T11:19:02.772211+0000","last_clean":"2023-10-11T11:19:02.772211+0000","last_became_active":"2023-10-11T10:43:41.772290+0000","last_became_peered":"2023-10-11T10:43:41.772290+0000","last_unstale":"2023-10-11T11:19:02.772211+0000","last_undegraded":"2023-10-11T11:19:02.772211+0000","last_fullsized":"2023-10-11T11:19:02.772211+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:26:06.909963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":647,"reported_epoch":499,"state":"active+undersized+degraded","last_fresh":"2023-10-11T11:19:00.625113+0000","last_change":"2023-10-11T11:15:31.654022+0000","last_active":"2023-10-11T11:19:00.625113+0000","last_peered":"2023-10-11T11:19:00.625113+0000","last_clean":"2023-10-11T11:15:27.636884+0000","last_became_active":"2023-10-11T11:15:31.654022+0000","last_became_peered":"2023-10-11T11:15:31.654022+0000","last_unstale":"2023-10-11T11:19:00.625113+0000","last_undegraded":"2023-10-11T11:15:31.646946+0000","last_fullsized":"2023-10-11T11:15:31.646751+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:36.739637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:36.739637+0000","last_clean_scrub_stamp":"2023-10-11T10:43:36.739637+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T11:33:33.696447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":519,"reported_epoch":499,"state":"active+clean+remapped","last_fresh":"2023-10-11T11:19:01.882635+0000","last_change":"2023-10-11T11:15:33.660200+0000","last_active":"2023-10-11T11:19:01.882635+0000","last_peered":"2023-10-11T11:19:01.882635+0000","last_clean":"2023-10-11T11:19:01.882635+0000","last_became_active":"2023-10-11T11:15:33.659619+0000","last_became_peered":"2023-10-11T11:15:33.659619+0000","last_unstale":"2023-10-11T11:19:01.882635+0000","last_undegraded":"2023-10-11T11:19:01.882635+0000","last_fullsized":"2023-10-11T11:19:01.882635+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T17:21:16.866392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":516,"reported_epoch":499,"state":"active+undersized","last_fresh":"2023-10-11T11:19:00.633628+0000","last_change":"2023-10-11T11:15:31.670813+0000","last_active":"2023-10-11T11:19:00.633628+0000","last_peered":"2023-10-11T11:19:00.633628+0000","last_clean":"2023-10-11T11:15:27.638219+0000","last_became_active":"2023-10-11T11:15:31.670813+0000","last_became_peered":"2023-10-11T11:15:31.670813+0000","last_unstale":"2023-10-11T11:19:00.633628+0000","last_undegraded":"2023-10-11T11:19:00.633628+0000","last_fullsized":"2023-10-11T11:15:31.658785+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:41:50.872346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":516,"reported_epoch":499,"state":"active+undersized","last_fresh":"2023-10-11T11:19:00.631668+0000","last_change":"2023-10-11T11:15:31.666378+0000","last_active":"2023-10-11T11:19:00.631668+0000","last_peered":"2023-10-11T11:19:00.631668+0000","last_clean":"2023-10-11T11:15:27.638303+0000","last_became_active":"2023-10-11T11:15:31.666378+0000","last_became_peered":"2023-10-11T11:15:31.666378+0000","last_unstale":"2023-10-11T11:19:00.631668+0000","last_undegraded":"2023-10-11T11:19:00.631668+0000","last_fullsized":"2023-10-11T11:15:31.649186+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T15:35:19.331209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":516,"reported_epoch":499,"state":"active+undersized","last_fresh":"2023-10-11T11:19:00.636960+0000","last_change":"2023-10-11T11:15:31.679465+0000","last_active":"2023-10-11T11:19:00.636960+0000","last_peered":"2023-10-11T11:19:00.636960+0000","last_clean":"2023-10-11T11:15:27.638749+0000","last_became_active":"2023-10-11T11:15:31.679465+0000","last_became_peered":"2023-10-11T11:15:31.679465+0000","last_unstale":"2023-10-11T11:19:00.636960+0000","last_undegraded":"2023-10-11T11:19:00.636960+0000","last_fullsized":"2023-10-11T11:15:31.659135+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T16:03:42.656692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":11,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738798,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":785348,"kb_used_data":5812,"kb_used_omap":96,"kb_used_meta":779423,"kb_avail":93586492,"statfs":{"total":96636764160,"available":95832567808,"internally_reserved":0,"allocated":5951488,"data_stored":3106918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":98609,"internal_metadata":798129871},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 11:14:37 2023","interfaces":[{"interface":"back","average":{"1min":0.314,"5min":0.316,"15min":0.443},"min":{"1min":0.24399999999999999,"5min":0.14799999999999999,"15min":0.14599999999999999},"max":{"1min":0.48499999999999999,"5min":1.712,"15min":9.6400000000000006},"last":0.31},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.35499999999999998,"15min":0.41299999999999998},"min":{"1min":0.20899999999999999,"5min":0.17199999999999999,"15min":0.16400000000000001},"max":{"1min":3.8180000000000001,"5min":3.8180000000000001,"15min":6.3200000000000003},"last":0.26500000000000001}]},{"osd":1,"last update":"Wed Oct 11 11:18:39 2023","interfaces":[{"interface":"back","average":{"1min":0.27500000000000002,"5min":0.29299999999999998,"15min":0.32400000000000001},"min":{"1min":0.154,"5min":0.154,"15min":0.154},"max":{"1min":0.41999999999999998,"5min":0.59299999999999997,"15min":2.9540000000000002},"last":0.317},{"interface":"front","average":{"1min":0.307,"5min":0.35899999999999999,"15min":0.36499999999999999},"min":{"1min":0.18099999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.45100000000000001,"5min":3.2719999999999998,"15min":3.2719999999999998},"last":0.39500000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738756,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1484496,"kb_used_data":406792,"kb_used_omap":107,"kb_used_meta":1077588,"kb_avail":92887344,"statfs":{"total":96636764160,"available":95116640256,"internally_reserved":0,"allocated":416555008,"data_stored":413648921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110309,"internal_metadata":1103450395},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Oct 11 11:14:30 2023","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.35399999999999998,"15min":0.40600000000000003},"min":{"1min":0.18099999999999999,"5min":0.155,"15min":0.14899999999999999},"max":{"1min":1.4319999999999999,"5min":2.3460000000000001,"15min":6.593},"last":0.26200000000000001},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.39100000000000001,"15min":0.48199999999999998},"min":{"1min":0.189,"5min":0.189,"15min":0.183},"max":{"1min":2.2120000000000002,"5min":2.4089999999999998,"15min":6.5960000000000001},"last":0.316}]},{"osd":2,"last update":"Wed Oct 11 11:14:30 2023","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.35499999999999998,"15min":0.47099999999999997},"min":{"1min":0.20300000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":1.2490000000000001,"5min":2.665,"15min":8.4730000000000008},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.375,"15min":0.442},"min":{"1min":0.23200000000000001,"5min":0.20499999999999999,"15min":0.19600000000000001},"max":{"1min":1.28,"5min":2.6230000000000002,"15min":7.0129999999999999},"last":0.33600000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738799,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":826440,"kb_used_data":6392,"kb_used_omap":88,"kb_used_meta":819943,"kb_avail":93545400,"statfs":{"total":96636764160,"available":95790489600,"internally_reserved":0,"allocated":6545408,"data_stored":3697286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":90523,"internal_metadata":839622245},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 11:14:33 2023","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.42799999999999999,"15min":0.41099999999999998},"min":{"1min":0.151,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":1.2809999999999999,"5min":3.5739999999999998,"15min":8.2479999999999993},"last":0.223},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.48299999999999998,"15min":0.49199999999999999},"min":{"1min":0.20699999999999999,"5min":0.182,"15min":0.17999999999999999},"max":{"1min":1.323,"5min":3.6080000000000001,"15min":8.2870000000000008},"last":0.28399999999999997}]},{"osd":2,"last update":"Wed Oct 11 11:18:40 2023","interfaces":[{"interface":"back","average":{"1min":0.33400000000000002,"5min":0.378,"15min":0.35599999999999998},"min":{"1min":0.16500000000000001,"5min":0.14699999999999999,"15min":0.14699999999999999},"max":{"1min":1.244,"5min":7.4829999999999997,"15min":10.617000000000001},"last":0.312},{"interface":"front","average":{"1min":0.34599999999999997,"5min":0.39000000000000001,"15min":0.34999999999999998},"min":{"1min":0.20399999999999999,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.81100000000000005,"5min":7.5350000000000001,"15min":7.5350000000000001},"last":0.28100000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-11T11:19:13.736 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-10-11T11:19:13.736 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 517, 'reported_epoch': 500, 'state': 'active+undersized', 'last_fresh': '2023-10-11T11:19:02.772828+0000', 'last_change': '2023-10-11T11:15:31.661747+0000', 'last_active': '2023-10-11T11:19:02.772828+0000', 'last_peered': '2023-10-11T11:19:02.772828+0000', 'last_clean': '2023-10-11T11:15:27.638215+0000', 'last_became_active': '2023-10-11T11:15:31.661747+0000', 'last_became_peered': '2023-10-11T11:15:31.661747+0000', 'last_unstale': '2023-10-11T11:19:02.772828+0000', 'last_undegraded': '2023-10-11T11:19:02.772828+0000', 'last_fullsized': '2023-10-11T11:15:31.649336+0000', 'mapping_epoch': 480, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_clean_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-12T22:15:25.047039+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-10-11T11:19:13.737 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-10-11T11:19:13.737 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 28, 'reported_epoch': 499, 'state': 'active+undersized', 'last_fresh': '2023-10-11T11:19:00.628569+0000', 'last_change': '2023-10-11T11:15:31.671363+0000', 'last_active': '2023-10-11T11:19:00.628569+0000', 'last_peered': '2023-10-11T11:19:00.628569+0000', 'last_clean': '2023-10-11T10:43:40.751300+0000', 'last_became_active': '2023-10-11T11:15:31.671363+0000', 'last_became_peered': '2023-10-11T11:15:31.671363+0000', 'last_unstale': '2023-10-11T11:19:00.628569+0000', 'last_undegraded': '2023-10-11T11:19:00.628569+0000', 'last_fullsized': '2023-10-11T11:15:31.652001+0000', 'mapping_epoch': 480, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_clean_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-12T21:01:53.884964+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-10-11T11:19:13.737 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-10-11T11:19:13.737 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "11'97", 'reported_seq': 647, 'reported_epoch': 499, 'state': 'active+undersized+degraded', 'last_fresh': '2023-10-11T11:19:00.625113+0000', 'last_change': '2023-10-11T11:15:31.654022+0000', 'last_active': '2023-10-11T11:19:00.625113+0000', 'last_peered': '2023-10-11T11:19:00.625113+0000', 'last_clean': '2023-10-11T11:15:27.636884+0000', 'last_became_active': '2023-10-11T11:15:31.654022+0000', 'last_became_peered': '2023-10-11T11:15:31.654022+0000', 'last_unstale': '2023-10-11T11:19:00.625113+0000', 'last_undegraded': '2023-10-11T11:15:31.646946+0000', 'last_fullsized': '2023-10-11T11:15:31.646751+0000', 'mapping_epoch': 480, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-11T10:43:36.739637+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-11T10:43:36.739637+0000', 'last_clean_scrub_stamp': '2023-10-11T10:43:36.739637+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-12T11:33:33.696447+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-10-11T11:19:13.738 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.0 is not active+clean 2023-10-11T11:19:13.738 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 519, 'reported_epoch': 499, 'state': 'active+clean+remapped', 'last_fresh': '2023-10-11T11:19:01.882635+0000', 'last_change': '2023-10-11T11:15:33.660200+0000', 'last_active': '2023-10-11T11:19:01.882635+0000', 'last_peered': '2023-10-11T11:19:01.882635+0000', 'last_clean': '2023-10-11T11:19:01.882635+0000', 'last_became_active': '2023-10-11T11:15:33.659619+0000', 'last_became_peered': '2023-10-11T11:15:33.659619+0000', 'last_unstale': '2023-10-11T11:19:01.882635+0000', 'last_undegraded': '2023-10-11T11:19:01.882635+0000', 'last_fullsized': '2023-10-11T11:19:01.882635+0000', 'mapping_epoch': 482, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 483, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_clean_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-12T17:21:16.866392+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2, 1], 'avail_no_missing': ['2', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-10-11T11:19:13.739 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-10-11T11:19:13.739 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 516, 'reported_epoch': 499, 'state': 'active+undersized', 'last_fresh': '2023-10-11T11:19:00.633628+0000', 'last_change': '2023-10-11T11:15:31.670813+0000', 'last_active': '2023-10-11T11:19:00.633628+0000', 'last_peered': '2023-10-11T11:19:00.633628+0000', 'last_clean': '2023-10-11T11:15:27.638219+0000', 'last_became_active': '2023-10-11T11:15:31.670813+0000', 'last_became_peered': '2023-10-11T11:15:31.670813+0000', 'last_unstale': '2023-10-11T11:19:00.633628+0000', 'last_undegraded': '2023-10-11T11:19:00.633628+0000', 'last_fullsized': '2023-10-11T11:15:31.658785+0000', 'mapping_epoch': 480, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_clean_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-12T21:41:50.872346+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-10-11T11:19:13.739 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-10-11T11:19:13.739 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 516, 'reported_epoch': 499, 'state': 'active+undersized', 'last_fresh': '2023-10-11T11:19:00.631668+0000', 'last_change': '2023-10-11T11:15:31.666378+0000', 'last_active': '2023-10-11T11:19:00.631668+0000', 'last_peered': '2023-10-11T11:19:00.631668+0000', 'last_clean': '2023-10-11T11:15:27.638303+0000', 'last_became_active': '2023-10-11T11:15:31.666378+0000', 'last_became_peered': '2023-10-11T11:15:31.666378+0000', 'last_unstale': '2023-10-11T11:19:00.631668+0000', 'last_undegraded': '2023-10-11T11:19:00.631668+0000', 'last_fullsized': '2023-10-11T11:15:31.649186+0000', 'mapping_epoch': 480, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_clean_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-12T15:35:19.331209+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-10-11T11:19:13.740 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-10-11T11:19:13.740 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 516, 'reported_epoch': 499, 'state': 'active+undersized', 'last_fresh': '2023-10-11T11:19:00.636960+0000', 'last_change': '2023-10-11T11:15:31.679465+0000', 'last_active': '2023-10-11T11:19:00.636960+0000', 'last_peered': '2023-10-11T11:19:00.636960+0000', 'last_clean': '2023-10-11T11:15:27.638749+0000', 'last_became_active': '2023-10-11T11:15:31.679465+0000', 'last_became_peered': '2023-10-11T11:15:31.679465+0000', 'last_unstale': '2023-10-11T11:19:00.636960+0000', 'last_undegraded': '2023-10-11T11:19:00.636960+0000', 'last_fullsized': '2023-10-11T11:15:31.659135+0000', 'mapping_epoch': 480, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_clean_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-12T16:03:42.656692+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-10-11T11:19:13.741 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-10-11T11:19:14.139 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T11:19:14.139 INFO:teuthology.orchestra.run.smithi111.stdout:{"fsid":"022ce6f2-9a5b-45ad-8b77-a1332d92f228","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2149,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":500,"num_osds":3,"num_up_osds":2,"osd_up_since":1697022930,"num_in_osds":3,"osd_in_since":1697021005,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":2},{"state_name":"active+undersized+degraded","count":1},{"state_name":"active+clean+remapped","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3170594816,"bytes_avail":286739697664,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":36,"modified":"2023-10-11T11:19:04.644408+0000","services":{}},"progress_events":{}} 2023-10-11T11:19:14.153 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-10-11T11:19:14.153 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-10-11T11:19:14.468 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T11:19:14.470 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2023-10-11T11:19:14.482 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":1567,"stamp":"2023-10-11T11:19:12.646621+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":11,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":107,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3096284,"kb_used_data":418996,"kb_used_omap":292,"kb_used_meta":2676955,"kb_avail":280019236,"statfs":{"total":289910292480,"available":286739697664,"internally_reserved":0,"allocated":429051904,"data_stored":420453125,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299441,"internal_metadata":2741202511},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19,"num_objects":-8,"num_object_clones":0,"num_object_copies":-16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-8,"num_whiteouts":0,"num_read":-2040,"num_read_kb":-1943,"num_write":-383,"num_write_kb":-303,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002922"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":517,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-10-11T11:19:02.772828+0000","last_change":"2023-10-11T11:15:31.661747+0000","last_active":"2023-10-11T11:19:02.772828+0000","last_peered":"2023-10-11T11:19:02.772828+0000","last_clean":"2023-10-11T11:15:27.638215+0000","last_became_active":"2023-10-11T11:15:31.661747+0000","last_became_peered":"2023-10-11T11:15:31.661747+0000","last_unstale":"2023-10-11T11:19:02.772828+0000","last_undegraded":"2023-10-11T11:19:02.772828+0000","last_fullsized":"2023-10-11T11:15:31.649336+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:15:25.047039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":28,"reported_epoch":499,"state":"active+undersized","last_fresh":"2023-10-11T11:19:00.628569+0000","last_change":"2023-10-11T11:15:31.671363+0000","last_active":"2023-10-11T11:19:00.628569+0000","last_peered":"2023-10-11T11:19:00.628569+0000","last_clean":"2023-10-11T10:43:40.751300+0000","last_became_active":"2023-10-11T11:15:31.671363+0000","last_became_peered":"2023-10-11T11:15:31.671363+0000","last_unstale":"2023-10-11T11:19:00.628569+0000","last_undegraded":"2023-10-11T11:19:00.628569+0000","last_fullsized":"2023-10-11T11:15:31.652001+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:01:53.884964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":512,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-10-11T11:19:03.429640+0000","last_change":"2023-10-11T10:43:41.772781+0000","last_active":"2023-10-11T11:19:03.429640+0000","last_peered":"2023-10-11T11:19:03.429640+0000","last_clean":"2023-10-11T11:19:03.429640+0000","last_became_active":"2023-10-11T10:43:41.772443+0000","last_became_peered":"2023-10-11T10:43:41.772443+0000","last_unstale":"2023-10-11T11:19:03.429640+0000","last_undegraded":"2023-10-11T11:19:03.429640+0000","last_fullsized":"2023-10-11T11:19:03.429640+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T14:15:30.547284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":513,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-10-11T11:19:02.772211+0000","last_change":"2023-10-11T10:43:41.772615+0000","last_active":"2023-10-11T11:19:02.772211+0000","last_peered":"2023-10-11T11:19:02.772211+0000","last_clean":"2023-10-11T11:19:02.772211+0000","last_became_active":"2023-10-11T10:43:41.772290+0000","last_became_peered":"2023-10-11T10:43:41.772290+0000","last_unstale":"2023-10-11T11:19:02.772211+0000","last_undegraded":"2023-10-11T11:19:02.772211+0000","last_fullsized":"2023-10-11T11:19:02.772211+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:26:06.909963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":647,"reported_epoch":499,"state":"active+undersized+degraded","last_fresh":"2023-10-11T11:19:00.625113+0000","last_change":"2023-10-11T11:15:31.654022+0000","last_active":"2023-10-11T11:19:00.625113+0000","last_peered":"2023-10-11T11:19:00.625113+0000","last_clean":"2023-10-11T11:15:27.636884+0000","last_became_active":"2023-10-11T11:15:31.654022+0000","last_became_peered":"2023-10-11T11:15:31.654022+0000","last_unstale":"2023-10-11T11:19:00.625113+0000","last_undegraded":"2023-10-11T11:15:31.646946+0000","last_fullsized":"2023-10-11T11:15:31.646751+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:36.739637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:36.739637+0000","last_clean_scrub_stamp":"2023-10-11T10:43:36.739637+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T11:33:33.696447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":519,"reported_epoch":499,"state":"active+clean+remapped","last_fresh":"2023-10-11T11:19:01.882635+0000","last_change":"2023-10-11T11:15:33.660200+0000","last_active":"2023-10-11T11:19:01.882635+0000","last_peered":"2023-10-11T11:19:01.882635+0000","last_clean":"2023-10-11T11:19:01.882635+0000","last_became_active":"2023-10-11T11:15:33.659619+0000","last_became_peered":"2023-10-11T11:15:33.659619+0000","last_unstale":"2023-10-11T11:19:01.882635+0000","last_undegraded":"2023-10-11T11:19:01.882635+0000","last_fullsized":"2023-10-11T11:19:01.882635+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T17:21:16.866392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":516,"reported_epoch":499,"state":"active+undersized","last_fresh":"2023-10-11T11:19:00.633628+0000","last_change":"2023-10-11T11:15:31.670813+0000","last_active":"2023-10-11T11:19:00.633628+0000","last_peered":"2023-10-11T11:19:00.633628+0000","last_clean":"2023-10-11T11:15:27.638219+0000","last_became_active":"2023-10-11T11:15:31.670813+0000","last_became_peered":"2023-10-11T11:15:31.670813+0000","last_unstale":"2023-10-11T11:19:00.633628+0000","last_undegraded":"2023-10-11T11:19:00.633628+0000","last_fullsized":"2023-10-11T11:15:31.658785+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:41:50.872346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":516,"reported_epoch":499,"state":"active+undersized","last_fresh":"2023-10-11T11:19:00.631668+0000","last_change":"2023-10-11T11:15:31.666378+0000","last_active":"2023-10-11T11:19:00.631668+0000","last_peered":"2023-10-11T11:19:00.631668+0000","last_clean":"2023-10-11T11:15:27.638303+0000","last_became_active":"2023-10-11T11:15:31.666378+0000","last_became_peered":"2023-10-11T11:15:31.666378+0000","last_unstale":"2023-10-11T11:19:00.631668+0000","last_undegraded":"2023-10-11T11:19:00.631668+0000","last_fullsized":"2023-10-11T11:15:31.649186+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T15:35:19.331209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":516,"reported_epoch":499,"state":"active+undersized","last_fresh":"2023-10-11T11:19:00.636960+0000","last_change":"2023-10-11T11:15:31.679465+0000","last_active":"2023-10-11T11:19:00.636960+0000","last_peered":"2023-10-11T11:19:00.636960+0000","last_clean":"2023-10-11T11:15:27.638749+0000","last_became_active":"2023-10-11T11:15:31.679465+0000","last_became_peered":"2023-10-11T11:15:31.679465+0000","last_unstale":"2023-10-11T11:19:00.636960+0000","last_undegraded":"2023-10-11T11:19:00.636960+0000","last_fullsized":"2023-10-11T11:15:31.659135+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T16:03:42.656692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":11,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738798,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":785348,"kb_used_data":5812,"kb_used_omap":96,"kb_used_meta":779423,"kb_avail":93586492,"statfs":{"total":96636764160,"available":95832567808,"internally_reserved":0,"allocated":5951488,"data_stored":3106918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":98609,"internal_metadata":798129871},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 11:14:37 2023","interfaces":[{"interface":"back","average":{"1min":0.314,"5min":0.316,"15min":0.443},"min":{"1min":0.24399999999999999,"5min":0.14799999999999999,"15min":0.14599999999999999},"max":{"1min":0.48499999999999999,"5min":1.712,"15min":9.6400000000000006},"last":0.31},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.35499999999999998,"15min":0.41299999999999998},"min":{"1min":0.20899999999999999,"5min":0.17199999999999999,"15min":0.16400000000000001},"max":{"1min":3.8180000000000001,"5min":3.8180000000000001,"15min":6.3200000000000003},"last":0.26500000000000001}]},{"osd":1,"last update":"Wed Oct 11 11:18:39 2023","interfaces":[{"interface":"back","average":{"1min":0.27500000000000002,"5min":0.29299999999999998,"15min":0.32400000000000001},"min":{"1min":0.154,"5min":0.154,"15min":0.154},"max":{"1min":0.41999999999999998,"5min":0.59299999999999997,"15min":2.9540000000000002},"last":0.317},{"interface":"front","average":{"1min":0.307,"5min":0.35899999999999999,"15min":0.36499999999999999},"min":{"1min":0.18099999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.45100000000000001,"5min":3.2719999999999998,"15min":3.2719999999999998},"last":0.39500000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738756,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1484496,"kb_used_data":406792,"kb_used_omap":107,"kb_used_meta":1077588,"kb_avail":92887344,"statfs":{"total":96636764160,"available":95116640256,"internally_reserved":0,"allocated":416555008,"data_stored":413648921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110309,"internal_metadata":1103450395},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Oct 11 11:14:30 2023","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.35399999999999998,"15min":0.40600000000000003},"min":{"1min":0.18099999999999999,"5min":0.155,"15min":0.14899999999999999},"max":{"1min":1.4319999999999999,"5min":2.3460000000000001,"15min":6.593},"last":0.26200000000000001},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.39100000000000001,"15min":0.48199999999999998},"min":{"1min":0.189,"5min":0.189,"15min":0.183},"max":{"1min":2.2120000000000002,"5min":2.4089999999999998,"15min":6.5960000000000001},"last":0.316}]},{"osd":2,"last update":"Wed Oct 11 11:14:30 2023","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.35499999999999998,"15min":0.47099999999999997},"min":{"1min":0.20300000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":1.2490000000000001,"5min":2.665,"15min":8.4730000000000008},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.375,"15min":0.442},"min":{"1min":0.23200000000000001,"5min":0.20499999999999999,"15min":0.19600000000000001},"max":{"1min":1.28,"5min":2.6230000000000002,"15min":7.0129999999999999},"last":0.33600000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738799,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":826440,"kb_used_data":6392,"kb_used_omap":88,"kb_used_meta":819943,"kb_avail":93545400,"statfs":{"total":96636764160,"available":95790489600,"internally_reserved":0,"allocated":6545408,"data_stored":3697286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":90523,"internal_metadata":839622245},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 11:14:33 2023","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.42799999999999999,"15min":0.41099999999999998},"min":{"1min":0.151,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":1.2809999999999999,"5min":3.5739999999999998,"15min":8.2479999999999993},"last":0.223},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.48299999999999998,"15min":0.49199999999999999},"min":{"1min":0.20699999999999999,"5min":0.182,"15min":0.17999999999999999},"max":{"1min":1.323,"5min":3.6080000000000001,"15min":8.2870000000000008},"last":0.28399999999999997}]},{"osd":2,"last update":"Wed Oct 11 11:18:40 2023","interfaces":[{"interface":"back","average":{"1min":0.33400000000000002,"5min":0.378,"15min":0.35599999999999998},"min":{"1min":0.16500000000000001,"5min":0.14699999999999999,"15min":0.14699999999999999},"max":{"1min":1.244,"5min":7.4829999999999997,"15min":10.617000000000001},"last":0.312},{"interface":"front","average":{"1min":0.34599999999999997,"5min":0.39000000000000001,"15min":0.34999999999999998},"min":{"1min":0.20399999999999999,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.81100000000000005,"5min":7.5350000000000001,"15min":7.5350000000000001},"last":0.28100000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-11T11:19:17.483 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-10-11T11:19:17.811 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T11:19:17.813 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2023-10-11T11:19:17.825 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":1569,"stamp":"2023-10-11T11:19:16.647588+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":11,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":107,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3096284,"kb_used_data":418996,"kb_used_omap":292,"kb_used_meta":2676955,"kb_avail":280019236,"statfs":{"total":289910292480,"available":286739697664,"internally_reserved":0,"allocated":429051904,"data_stored":420453125,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299441,"internal_metadata":2741202511},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003210"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":517,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-10-11T11:19:02.772828+0000","last_change":"2023-10-11T11:15:31.661747+0000","last_active":"2023-10-11T11:19:02.772828+0000","last_peered":"2023-10-11T11:19:02.772828+0000","last_clean":"2023-10-11T11:15:27.638215+0000","last_became_active":"2023-10-11T11:15:31.661747+0000","last_became_peered":"2023-10-11T11:15:31.661747+0000","last_unstale":"2023-10-11T11:19:02.772828+0000","last_undegraded":"2023-10-11T11:19:02.772828+0000","last_fullsized":"2023-10-11T11:15:31.649336+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:15:25.047039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":28,"reported_epoch":499,"state":"active+undersized","last_fresh":"2023-10-11T11:19:00.628569+0000","last_change":"2023-10-11T11:15:31.671363+0000","last_active":"2023-10-11T11:19:00.628569+0000","last_peered":"2023-10-11T11:19:00.628569+0000","last_clean":"2023-10-11T10:43:40.751300+0000","last_became_active":"2023-10-11T11:15:31.671363+0000","last_became_peered":"2023-10-11T11:15:31.671363+0000","last_unstale":"2023-10-11T11:19:00.628569+0000","last_undegraded":"2023-10-11T11:19:00.628569+0000","last_fullsized":"2023-10-11T11:15:31.652001+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:01:53.884964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":512,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-10-11T11:19:03.429640+0000","last_change":"2023-10-11T10:43:41.772781+0000","last_active":"2023-10-11T11:19:03.429640+0000","last_peered":"2023-10-11T11:19:03.429640+0000","last_clean":"2023-10-11T11:19:03.429640+0000","last_became_active":"2023-10-11T10:43:41.772443+0000","last_became_peered":"2023-10-11T10:43:41.772443+0000","last_unstale":"2023-10-11T11:19:03.429640+0000","last_undegraded":"2023-10-11T11:19:03.429640+0000","last_fullsized":"2023-10-11T11:19:03.429640+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T14:15:30.547284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":513,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-10-11T11:19:02.772211+0000","last_change":"2023-10-11T10:43:41.772615+0000","last_active":"2023-10-11T11:19:02.772211+0000","last_peered":"2023-10-11T11:19:02.772211+0000","last_clean":"2023-10-11T11:19:02.772211+0000","last_became_active":"2023-10-11T10:43:41.772290+0000","last_became_peered":"2023-10-11T10:43:41.772290+0000","last_unstale":"2023-10-11T11:19:02.772211+0000","last_undegraded":"2023-10-11T11:19:02.772211+0000","last_fullsized":"2023-10-11T11:19:02.772211+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:26:06.909963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":647,"reported_epoch":499,"state":"active+undersized+degraded","last_fresh":"2023-10-11T11:19:00.625113+0000","last_change":"2023-10-11T11:15:31.654022+0000","last_active":"2023-10-11T11:19:00.625113+0000","last_peered":"2023-10-11T11:19:00.625113+0000","last_clean":"2023-10-11T11:15:27.636884+0000","last_became_active":"2023-10-11T11:15:31.654022+0000","last_became_peered":"2023-10-11T11:15:31.654022+0000","last_unstale":"2023-10-11T11:19:00.625113+0000","last_undegraded":"2023-10-11T11:15:31.646946+0000","last_fullsized":"2023-10-11T11:15:31.646751+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:36.739637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:36.739637+0000","last_clean_scrub_stamp":"2023-10-11T10:43:36.739637+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T11:33:33.696447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":519,"reported_epoch":499,"state":"active+clean+remapped","last_fresh":"2023-10-11T11:19:01.882635+0000","last_change":"2023-10-11T11:15:33.660200+0000","last_active":"2023-10-11T11:19:01.882635+0000","last_peered":"2023-10-11T11:19:01.882635+0000","last_clean":"2023-10-11T11:19:01.882635+0000","last_became_active":"2023-10-11T11:15:33.659619+0000","last_became_peered":"2023-10-11T11:15:33.659619+0000","last_unstale":"2023-10-11T11:19:01.882635+0000","last_undegraded":"2023-10-11T11:19:01.882635+0000","last_fullsized":"2023-10-11T11:19:01.882635+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T17:21:16.866392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":517,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-10-11T11:19:02.775128+0000","last_change":"2023-10-11T11:15:31.670813+0000","last_active":"2023-10-11T11:19:02.775128+0000","last_peered":"2023-10-11T11:19:02.775128+0000","last_clean":"2023-10-11T11:15:27.638219+0000","last_became_active":"2023-10-11T11:15:31.670813+0000","last_became_peered":"2023-10-11T11:15:31.670813+0000","last_unstale":"2023-10-11T11:19:02.775128+0000","last_undegraded":"2023-10-11T11:19:02.775128+0000","last_fullsized":"2023-10-11T11:15:31.658785+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:41:50.872346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":516,"reported_epoch":499,"state":"active+undersized","last_fresh":"2023-10-11T11:19:00.631668+0000","last_change":"2023-10-11T11:15:31.666378+0000","last_active":"2023-10-11T11:19:00.631668+0000","last_peered":"2023-10-11T11:19:00.631668+0000","last_clean":"2023-10-11T11:15:27.638303+0000","last_became_active":"2023-10-11T11:15:31.666378+0000","last_became_peered":"2023-10-11T11:15:31.666378+0000","last_unstale":"2023-10-11T11:19:00.631668+0000","last_undegraded":"2023-10-11T11:19:00.631668+0000","last_fullsized":"2023-10-11T11:15:31.649186+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T15:35:19.331209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":516,"reported_epoch":499,"state":"active+undersized","last_fresh":"2023-10-11T11:19:00.636960+0000","last_change":"2023-10-11T11:15:31.679465+0000","last_active":"2023-10-11T11:19:00.636960+0000","last_peered":"2023-10-11T11:19:00.636960+0000","last_clean":"2023-10-11T11:15:27.638749+0000","last_became_active":"2023-10-11T11:15:31.679465+0000","last_became_peered":"2023-10-11T11:15:31.679465+0000","last_unstale":"2023-10-11T11:19:00.636960+0000","last_undegraded":"2023-10-11T11:19:00.636960+0000","last_fullsized":"2023-10-11T11:15:31.659135+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T16:03:42.656692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":11,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738799,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":785348,"kb_used_data":5812,"kb_used_omap":96,"kb_used_meta":779423,"kb_avail":93586492,"statfs":{"total":96636764160,"available":95832567808,"internally_reserved":0,"allocated":5951488,"data_stored":3106918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":98609,"internal_metadata":798129871},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 11:14:37 2023","interfaces":[{"interface":"back","average":{"1min":0.314,"5min":0.316,"15min":0.443},"min":{"1min":0.24399999999999999,"5min":0.14799999999999999,"15min":0.14599999999999999},"max":{"1min":0.48499999999999999,"5min":1.712,"15min":9.6400000000000006},"last":0.31},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.35499999999999998,"15min":0.41299999999999998},"min":{"1min":0.20899999999999999,"5min":0.17199999999999999,"15min":0.16400000000000001},"max":{"1min":3.8180000000000001,"5min":3.8180000000000001,"15min":6.3200000000000003},"last":0.26500000000000001}]},{"osd":1,"last update":"Wed Oct 11 11:18:39 2023","interfaces":[{"interface":"back","average":{"1min":0.27500000000000002,"5min":0.29299999999999998,"15min":0.32400000000000001},"min":{"1min":0.154,"5min":0.154,"15min":0.154},"max":{"1min":0.41999999999999998,"5min":0.59299999999999997,"15min":2.9540000000000002},"last":0.39500000000000002},{"interface":"front","average":{"1min":0.307,"5min":0.35899999999999999,"15min":0.36499999999999999},"min":{"1min":0.18099999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.45100000000000001,"5min":3.2719999999999998,"15min":3.2719999999999998},"last":0.42599999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738756,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1484496,"kb_used_data":406792,"kb_used_omap":107,"kb_used_meta":1077588,"kb_avail":92887344,"statfs":{"total":96636764160,"available":95116640256,"internally_reserved":0,"allocated":416555008,"data_stored":413648921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110309,"internal_metadata":1103450395},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Oct 11 11:14:30 2023","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.35399999999999998,"15min":0.40600000000000003},"min":{"1min":0.18099999999999999,"5min":0.155,"15min":0.14899999999999999},"max":{"1min":1.4319999999999999,"5min":2.3460000000000001,"15min":6.593},"last":0.26200000000000001},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.39100000000000001,"15min":0.48199999999999998},"min":{"1min":0.189,"5min":0.189,"15min":0.183},"max":{"1min":2.2120000000000002,"5min":2.4089999999999998,"15min":6.5960000000000001},"last":0.316}]},{"osd":2,"last update":"Wed Oct 11 11:14:30 2023","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.35499999999999998,"15min":0.47099999999999997},"min":{"1min":0.20300000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":1.2490000000000001,"5min":2.665,"15min":8.4730000000000008},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.375,"15min":0.442},"min":{"1min":0.23200000000000001,"5min":0.20499999999999999,"15min":0.19600000000000001},"max":{"1min":1.28,"5min":2.6230000000000002,"15min":7.0129999999999999},"last":0.33600000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738800,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":826440,"kb_used_data":6392,"kb_used_omap":88,"kb_used_meta":819943,"kb_avail":93545400,"statfs":{"total":96636764160,"available":95790489600,"internally_reserved":0,"allocated":6545408,"data_stored":3697286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":90523,"internal_metadata":839622245},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 11:14:33 2023","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.42799999999999999,"15min":0.41099999999999998},"min":{"1min":0.151,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":1.2809999999999999,"5min":3.5739999999999998,"15min":8.2479999999999993},"last":0.223},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.48299999999999998,"15min":0.49199999999999999},"min":{"1min":0.20699999999999999,"5min":0.182,"15min":0.17999999999999999},"max":{"1min":1.323,"5min":3.6080000000000001,"15min":8.2870000000000008},"last":0.28399999999999997}]},{"osd":2,"last update":"Wed Oct 11 11:18:40 2023","interfaces":[{"interface":"back","average":{"1min":0.33400000000000002,"5min":0.378,"15min":0.35599999999999998},"min":{"1min":0.16500000000000001,"5min":0.14699999999999999,"15min":0.14699999999999999},"max":{"1min":1.244,"5min":7.4829999999999997,"15min":10.617000000000001},"last":0.32000000000000001},{"interface":"front","average":{"1min":0.34599999999999997,"5min":0.39000000000000001,"15min":0.34999999999999998},"min":{"1min":0.20399999999999999,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.81100000000000005,"5min":7.5350000000000001,"15min":7.5350000000000001},"last":0.36199999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-11T11:19:17.826 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-10-11T11:19:18.140 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T11:19:18.142 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2023-10-11T11:19:18.153 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":1569,"stamp":"2023-10-11T11:19:16.647588+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":11,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":107,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3096284,"kb_used_data":418996,"kb_used_omap":292,"kb_used_meta":2676955,"kb_avail":280019236,"statfs":{"total":289910292480,"available":286739697664,"internally_reserved":0,"allocated":429051904,"data_stored":420453125,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299441,"internal_metadata":2741202511},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003210"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":517,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-10-11T11:19:02.772828+0000","last_change":"2023-10-11T11:15:31.661747+0000","last_active":"2023-10-11T11:19:02.772828+0000","last_peered":"2023-10-11T11:19:02.772828+0000","last_clean":"2023-10-11T11:15:27.638215+0000","last_became_active":"2023-10-11T11:15:31.661747+0000","last_became_peered":"2023-10-11T11:15:31.661747+0000","last_unstale":"2023-10-11T11:19:02.772828+0000","last_undegraded":"2023-10-11T11:19:02.772828+0000","last_fullsized":"2023-10-11T11:15:31.649336+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:15:25.047039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":28,"reported_epoch":499,"state":"active+undersized","last_fresh":"2023-10-11T11:19:00.628569+0000","last_change":"2023-10-11T11:15:31.671363+0000","last_active":"2023-10-11T11:19:00.628569+0000","last_peered":"2023-10-11T11:19:00.628569+0000","last_clean":"2023-10-11T10:43:40.751300+0000","last_became_active":"2023-10-11T11:15:31.671363+0000","last_became_peered":"2023-10-11T11:15:31.671363+0000","last_unstale":"2023-10-11T11:19:00.628569+0000","last_undegraded":"2023-10-11T11:19:00.628569+0000","last_fullsized":"2023-10-11T11:15:31.652001+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:01:53.884964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":512,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-10-11T11:19:03.429640+0000","last_change":"2023-10-11T10:43:41.772781+0000","last_active":"2023-10-11T11:19:03.429640+0000","last_peered":"2023-10-11T11:19:03.429640+0000","last_clean":"2023-10-11T11:19:03.429640+0000","last_became_active":"2023-10-11T10:43:41.772443+0000","last_became_peered":"2023-10-11T10:43:41.772443+0000","last_unstale":"2023-10-11T11:19:03.429640+0000","last_undegraded":"2023-10-11T11:19:03.429640+0000","last_fullsized":"2023-10-11T11:19:03.429640+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T14:15:30.547284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":513,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-10-11T11:19:02.772211+0000","last_change":"2023-10-11T10:43:41.772615+0000","last_active":"2023-10-11T11:19:02.772211+0000","last_peered":"2023-10-11T11:19:02.772211+0000","last_clean":"2023-10-11T11:19:02.772211+0000","last_became_active":"2023-10-11T10:43:41.772290+0000","last_became_peered":"2023-10-11T10:43:41.772290+0000","last_unstale":"2023-10-11T11:19:02.772211+0000","last_undegraded":"2023-10-11T11:19:02.772211+0000","last_fullsized":"2023-10-11T11:19:02.772211+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:26:06.909963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":647,"reported_epoch":499,"state":"active+undersized+degraded","last_fresh":"2023-10-11T11:19:00.625113+0000","last_change":"2023-10-11T11:15:31.654022+0000","last_active":"2023-10-11T11:19:00.625113+0000","last_peered":"2023-10-11T11:19:00.625113+0000","last_clean":"2023-10-11T11:15:27.636884+0000","last_became_active":"2023-10-11T11:15:31.654022+0000","last_became_peered":"2023-10-11T11:15:31.654022+0000","last_unstale":"2023-10-11T11:19:00.625113+0000","last_undegraded":"2023-10-11T11:15:31.646946+0000","last_fullsized":"2023-10-11T11:15:31.646751+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:36.739637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:36.739637+0000","last_clean_scrub_stamp":"2023-10-11T10:43:36.739637+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T11:33:33.696447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":519,"reported_epoch":499,"state":"active+clean+remapped","last_fresh":"2023-10-11T11:19:01.882635+0000","last_change":"2023-10-11T11:15:33.660200+0000","last_active":"2023-10-11T11:19:01.882635+0000","last_peered":"2023-10-11T11:19:01.882635+0000","last_clean":"2023-10-11T11:19:01.882635+0000","last_became_active":"2023-10-11T11:15:33.659619+0000","last_became_peered":"2023-10-11T11:15:33.659619+0000","last_unstale":"2023-10-11T11:19:01.882635+0000","last_undegraded":"2023-10-11T11:19:01.882635+0000","last_fullsized":"2023-10-11T11:19:01.882635+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T17:21:16.866392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":517,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-10-11T11:19:02.775128+0000","last_change":"2023-10-11T11:15:31.670813+0000","last_active":"2023-10-11T11:19:02.775128+0000","last_peered":"2023-10-11T11:19:02.775128+0000","last_clean":"2023-10-11T11:15:27.638219+0000","last_became_active":"2023-10-11T11:15:31.670813+0000","last_became_peered":"2023-10-11T11:15:31.670813+0000","last_unstale":"2023-10-11T11:19:02.775128+0000","last_undegraded":"2023-10-11T11:19:02.775128+0000","last_fullsized":"2023-10-11T11:15:31.658785+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:41:50.872346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":516,"reported_epoch":499,"state":"active+undersized","last_fresh":"2023-10-11T11:19:00.631668+0000","last_change":"2023-10-11T11:15:31.666378+0000","last_active":"2023-10-11T11:19:00.631668+0000","last_peered":"2023-10-11T11:19:00.631668+0000","last_clean":"2023-10-11T11:15:27.638303+0000","last_became_active":"2023-10-11T11:15:31.666378+0000","last_became_peered":"2023-10-11T11:15:31.666378+0000","last_unstale":"2023-10-11T11:19:00.631668+0000","last_undegraded":"2023-10-11T11:19:00.631668+0000","last_fullsized":"2023-10-11T11:15:31.649186+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T15:35:19.331209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":516,"reported_epoch":499,"state":"active+undersized","last_fresh":"2023-10-11T11:19:00.636960+0000","last_change":"2023-10-11T11:15:31.679465+0000","last_active":"2023-10-11T11:19:00.636960+0000","last_peered":"2023-10-11T11:19:00.636960+0000","last_clean":"2023-10-11T11:15:27.638749+0000","last_became_active":"2023-10-11T11:15:31.679465+0000","last_became_peered":"2023-10-11T11:15:31.679465+0000","last_unstale":"2023-10-11T11:19:00.636960+0000","last_undegraded":"2023-10-11T11:19:00.636960+0000","last_fullsized":"2023-10-11T11:15:31.659135+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T16:03:42.656692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":11,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738799,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":785348,"kb_used_data":5812,"kb_used_omap":96,"kb_used_meta":779423,"kb_avail":93586492,"statfs":{"total":96636764160,"available":95832567808,"internally_reserved":0,"allocated":5951488,"data_stored":3106918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":98609,"internal_metadata":798129871},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 11:14:37 2023","interfaces":[{"interface":"back","average":{"1min":0.314,"5min":0.316,"15min":0.443},"min":{"1min":0.24399999999999999,"5min":0.14799999999999999,"15min":0.14599999999999999},"max":{"1min":0.48499999999999999,"5min":1.712,"15min":9.6400000000000006},"last":0.31},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.35499999999999998,"15min":0.41299999999999998},"min":{"1min":0.20899999999999999,"5min":0.17199999999999999,"15min":0.16400000000000001},"max":{"1min":3.8180000000000001,"5min":3.8180000000000001,"15min":6.3200000000000003},"last":0.26500000000000001}]},{"osd":1,"last update":"Wed Oct 11 11:18:39 2023","interfaces":[{"interface":"back","average":{"1min":0.27500000000000002,"5min":0.29299999999999998,"15min":0.32400000000000001},"min":{"1min":0.154,"5min":0.154,"15min":0.154},"max":{"1min":0.41999999999999998,"5min":0.59299999999999997,"15min":2.9540000000000002},"last":0.39500000000000002},{"interface":"front","average":{"1min":0.307,"5min":0.35899999999999999,"15min":0.36499999999999999},"min":{"1min":0.18099999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.45100000000000001,"5min":3.2719999999999998,"15min":3.2719999999999998},"last":0.42599999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738756,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1484496,"kb_used_data":406792,"kb_used_omap":107,"kb_used_meta":1077588,"kb_avail":92887344,"statfs":{"total":96636764160,"available":95116640256,"internally_reserved":0,"allocated":416555008,"data_stored":413648921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110309,"internal_metadata":1103450395},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Oct 11 11:14:30 2023","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.35399999999999998,"15min":0.40600000000000003},"min":{"1min":0.18099999999999999,"5min":0.155,"15min":0.14899999999999999},"max":{"1min":1.4319999999999999,"5min":2.3460000000000001,"15min":6.593},"last":0.26200000000000001},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.39100000000000001,"15min":0.48199999999999998},"min":{"1min":0.189,"5min":0.189,"15min":0.183},"max":{"1min":2.2120000000000002,"5min":2.4089999999999998,"15min":6.5960000000000001},"last":0.316}]},{"osd":2,"last update":"Wed Oct 11 11:14:30 2023","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.35499999999999998,"15min":0.47099999999999997},"min":{"1min":0.20300000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":1.2490000000000001,"5min":2.665,"15min":8.4730000000000008},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.375,"15min":0.442},"min":{"1min":0.23200000000000001,"5min":0.20499999999999999,"15min":0.19600000000000001},"max":{"1min":1.28,"5min":2.6230000000000002,"15min":7.0129999999999999},"last":0.33600000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738800,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":826440,"kb_used_data":6392,"kb_used_omap":88,"kb_used_meta":819943,"kb_avail":93545400,"statfs":{"total":96636764160,"available":95790489600,"internally_reserved":0,"allocated":6545408,"data_stored":3697286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":90523,"internal_metadata":839622245},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 11:14:33 2023","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.42799999999999999,"15min":0.41099999999999998},"min":{"1min":0.151,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":1.2809999999999999,"5min":3.5739999999999998,"15min":8.2479999999999993},"last":0.223},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.48299999999999998,"15min":0.49199999999999999},"min":{"1min":0.20699999999999999,"5min":0.182,"15min":0.17999999999999999},"max":{"1min":1.323,"5min":3.6080000000000001,"15min":8.2870000000000008},"last":0.28399999999999997}]},{"osd":2,"last update":"Wed Oct 11 11:18:40 2023","interfaces":[{"interface":"back","average":{"1min":0.33400000000000002,"5min":0.378,"15min":0.35599999999999998},"min":{"1min":0.16500000000000001,"5min":0.14699999999999999,"15min":0.14699999999999999},"max":{"1min":1.244,"5min":7.4829999999999997,"15min":10.617000000000001},"last":0.32000000000000001},{"interface":"front","average":{"1min":0.34599999999999997,"5min":0.39000000000000001,"15min":0.34999999999999998},"min":{"1min":0.20399999999999999,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.81100000000000005,"5min":7.5350000000000001,"15min":7.5350000000000001},"last":0.36199999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-11T11:19:18.154 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-10-11T11:19:18.154 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 517, 'reported_epoch': 500, 'state': 'active+undersized', 'last_fresh': '2023-10-11T11:19:02.772828+0000', 'last_change': '2023-10-11T11:15:31.661747+0000', 'last_active': '2023-10-11T11:19:02.772828+0000', 'last_peered': '2023-10-11T11:19:02.772828+0000', 'last_clean': '2023-10-11T11:15:27.638215+0000', 'last_became_active': '2023-10-11T11:15:31.661747+0000', 'last_became_peered': '2023-10-11T11:15:31.661747+0000', 'last_unstale': '2023-10-11T11:19:02.772828+0000', 'last_undegraded': '2023-10-11T11:19:02.772828+0000', 'last_fullsized': '2023-10-11T11:15:31.649336+0000', 'mapping_epoch': 480, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_clean_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-12T22:15:25.047039+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-10-11T11:19:18.155 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-10-11T11:19:18.155 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 28, 'reported_epoch': 499, 'state': 'active+undersized', 'last_fresh': '2023-10-11T11:19:00.628569+0000', 'last_change': '2023-10-11T11:15:31.671363+0000', 'last_active': '2023-10-11T11:19:00.628569+0000', 'last_peered': '2023-10-11T11:19:00.628569+0000', 'last_clean': '2023-10-11T10:43:40.751300+0000', 'last_became_active': '2023-10-11T11:15:31.671363+0000', 'last_became_peered': '2023-10-11T11:15:31.671363+0000', 'last_unstale': '2023-10-11T11:19:00.628569+0000', 'last_undegraded': '2023-10-11T11:19:00.628569+0000', 'last_fullsized': '2023-10-11T11:15:31.652001+0000', 'mapping_epoch': 480, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_clean_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-12T21:01:53.884964+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-10-11T11:19:18.155 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-10-11T11:19:18.155 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "11'97", 'reported_seq': 647, 'reported_epoch': 499, 'state': 'active+undersized+degraded', 'last_fresh': '2023-10-11T11:19:00.625113+0000', 'last_change': '2023-10-11T11:15:31.654022+0000', 'last_active': '2023-10-11T11:19:00.625113+0000', 'last_peered': '2023-10-11T11:19:00.625113+0000', 'last_clean': '2023-10-11T11:15:27.636884+0000', 'last_became_active': '2023-10-11T11:15:31.654022+0000', 'last_became_peered': '2023-10-11T11:15:31.654022+0000', 'last_unstale': '2023-10-11T11:19:00.625113+0000', 'last_undegraded': '2023-10-11T11:15:31.646946+0000', 'last_fullsized': '2023-10-11T11:15:31.646751+0000', 'mapping_epoch': 480, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-11T10:43:36.739637+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-11T10:43:36.739637+0000', 'last_clean_scrub_stamp': '2023-10-11T10:43:36.739637+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-12T11:33:33.696447+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-10-11T11:19:18.156 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.0 is not active+clean 2023-10-11T11:19:18.156 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 519, 'reported_epoch': 499, 'state': 'active+clean+remapped', 'last_fresh': '2023-10-11T11:19:01.882635+0000', 'last_change': '2023-10-11T11:15:33.660200+0000', 'last_active': '2023-10-11T11:19:01.882635+0000', 'last_peered': '2023-10-11T11:19:01.882635+0000', 'last_clean': '2023-10-11T11:19:01.882635+0000', 'last_became_active': '2023-10-11T11:15:33.659619+0000', 'last_became_peered': '2023-10-11T11:15:33.659619+0000', 'last_unstale': '2023-10-11T11:19:01.882635+0000', 'last_undegraded': '2023-10-11T11:19:01.882635+0000', 'last_fullsized': '2023-10-11T11:19:01.882635+0000', 'mapping_epoch': 482, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 483, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_clean_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-12T17:21:16.866392+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2, 1], 'avail_no_missing': ['2', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-10-11T11:19:18.156 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-10-11T11:19:18.157 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 517, 'reported_epoch': 500, 'state': 'active+undersized', 'last_fresh': '2023-10-11T11:19:02.775128+0000', 'last_change': '2023-10-11T11:15:31.670813+0000', 'last_active': '2023-10-11T11:19:02.775128+0000', 'last_peered': '2023-10-11T11:19:02.775128+0000', 'last_clean': '2023-10-11T11:15:27.638219+0000', 'last_became_active': '2023-10-11T11:15:31.670813+0000', 'last_became_peered': '2023-10-11T11:15:31.670813+0000', 'last_unstale': '2023-10-11T11:19:02.775128+0000', 'last_undegraded': '2023-10-11T11:19:02.775128+0000', 'last_fullsized': '2023-10-11T11:15:31.658785+0000', 'mapping_epoch': 480, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_clean_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-12T21:41:50.872346+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-10-11T11:19:18.157 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-10-11T11:19:18.157 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 516, 'reported_epoch': 499, 'state': 'active+undersized', 'last_fresh': '2023-10-11T11:19:00.631668+0000', 'last_change': '2023-10-11T11:15:31.666378+0000', 'last_active': '2023-10-11T11:19:00.631668+0000', 'last_peered': '2023-10-11T11:19:00.631668+0000', 'last_clean': '2023-10-11T11:15:27.638303+0000', 'last_became_active': '2023-10-11T11:15:31.666378+0000', 'last_became_peered': '2023-10-11T11:15:31.666378+0000', 'last_unstale': '2023-10-11T11:19:00.631668+0000', 'last_undegraded': '2023-10-11T11:19:00.631668+0000', 'last_fullsized': '2023-10-11T11:15:31.649186+0000', 'mapping_epoch': 480, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_clean_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-12T15:35:19.331209+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-10-11T11:19:18.158 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-10-11T11:19:18.158 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 516, 'reported_epoch': 499, 'state': 'active+undersized', 'last_fresh': '2023-10-11T11:19:00.636960+0000', 'last_change': '2023-10-11T11:15:31.679465+0000', 'last_active': '2023-10-11T11:19:00.636960+0000', 'last_peered': '2023-10-11T11:19:00.636960+0000', 'last_clean': '2023-10-11T11:15:27.638749+0000', 'last_became_active': '2023-10-11T11:15:31.679465+0000', 'last_became_peered': '2023-10-11T11:15:31.679465+0000', 'last_unstale': '2023-10-11T11:19:00.636960+0000', 'last_undegraded': '2023-10-11T11:19:00.636960+0000', 'last_fullsized': '2023-10-11T11:15:31.659135+0000', 'mapping_epoch': 480, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_clean_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-12T16:03:42.656692+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-10-11T11:19:18.158 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-10-11T11:19:18.553 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T11:19:18.554 INFO:teuthology.orchestra.run.smithi111.stdout:{"fsid":"022ce6f2-9a5b-45ad-8b77-a1332d92f228","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2153,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":500,"num_osds":3,"num_up_osds":2,"osd_up_since":1697022930,"num_in_osds":3,"osd_in_since":1697021005,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":2},{"state_name":"active+undersized+degraded","count":1},{"state_name":"active+clean+remapped","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3170594816,"bytes_avail":286739697664,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":36,"modified":"2023-10-11T11:19:04.644408+0000","services":{}},"progress_events":{}} 2023-10-11T11:19:18.567 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-10-11T11:19:18.568 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-10-11T11:19:18.884 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T11:19:18.886 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2023-10-11T11:19:18.897 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":1570,"stamp":"2023-10-11T11:19:18.648179+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":11,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":107,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3096284,"kb_used_data":418996,"kb_used_omap":292,"kb_used_meta":2676955,"kb_avail":280019236,"statfs":{"total":289910292480,"available":286739697664,"internally_reserved":0,"allocated":429051904,"data_stored":420453125,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299441,"internal_metadata":2741202511},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003319"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":517,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-10-11T11:19:02.772828+0000","last_change":"2023-10-11T11:15:31.661747+0000","last_active":"2023-10-11T11:19:02.772828+0000","last_peered":"2023-10-11T11:19:02.772828+0000","last_clean":"2023-10-11T11:15:27.638215+0000","last_became_active":"2023-10-11T11:15:31.661747+0000","last_became_peered":"2023-10-11T11:15:31.661747+0000","last_unstale":"2023-10-11T11:19:02.772828+0000","last_undegraded":"2023-10-11T11:19:02.772828+0000","last_fullsized":"2023-10-11T11:15:31.649336+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:15:25.047039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":28,"reported_epoch":499,"state":"active+undersized","last_fresh":"2023-10-11T11:19:00.628569+0000","last_change":"2023-10-11T11:15:31.671363+0000","last_active":"2023-10-11T11:19:00.628569+0000","last_peered":"2023-10-11T11:19:00.628569+0000","last_clean":"2023-10-11T10:43:40.751300+0000","last_became_active":"2023-10-11T11:15:31.671363+0000","last_became_peered":"2023-10-11T11:15:31.671363+0000","last_unstale":"2023-10-11T11:19:00.628569+0000","last_undegraded":"2023-10-11T11:19:00.628569+0000","last_fullsized":"2023-10-11T11:15:31.652001+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:01:53.884964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":512,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-10-11T11:19:03.429640+0000","last_change":"2023-10-11T10:43:41.772781+0000","last_active":"2023-10-11T11:19:03.429640+0000","last_peered":"2023-10-11T11:19:03.429640+0000","last_clean":"2023-10-11T11:19:03.429640+0000","last_became_active":"2023-10-11T10:43:41.772443+0000","last_became_peered":"2023-10-11T10:43:41.772443+0000","last_unstale":"2023-10-11T11:19:03.429640+0000","last_undegraded":"2023-10-11T11:19:03.429640+0000","last_fullsized":"2023-10-11T11:19:03.429640+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T14:15:30.547284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":513,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-10-11T11:19:02.772211+0000","last_change":"2023-10-11T10:43:41.772615+0000","last_active":"2023-10-11T11:19:02.772211+0000","last_peered":"2023-10-11T11:19:02.772211+0000","last_clean":"2023-10-11T11:19:02.772211+0000","last_became_active":"2023-10-11T10:43:41.772290+0000","last_became_peered":"2023-10-11T10:43:41.772290+0000","last_unstale":"2023-10-11T11:19:02.772211+0000","last_undegraded":"2023-10-11T11:19:02.772211+0000","last_fullsized":"2023-10-11T11:19:02.772211+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:26:06.909963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":647,"reported_epoch":499,"state":"active+undersized+degraded","last_fresh":"2023-10-11T11:19:00.625113+0000","last_change":"2023-10-11T11:15:31.654022+0000","last_active":"2023-10-11T11:19:00.625113+0000","last_peered":"2023-10-11T11:19:00.625113+0000","last_clean":"2023-10-11T11:15:27.636884+0000","last_became_active":"2023-10-11T11:15:31.654022+0000","last_became_peered":"2023-10-11T11:15:31.654022+0000","last_unstale":"2023-10-11T11:19:00.625113+0000","last_undegraded":"2023-10-11T11:15:31.646946+0000","last_fullsized":"2023-10-11T11:15:31.646751+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:36.739637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:36.739637+0000","last_clean_scrub_stamp":"2023-10-11T10:43:36.739637+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T11:33:33.696447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":519,"reported_epoch":499,"state":"active+clean+remapped","last_fresh":"2023-10-11T11:19:01.882635+0000","last_change":"2023-10-11T11:15:33.660200+0000","last_active":"2023-10-11T11:19:01.882635+0000","last_peered":"2023-10-11T11:19:01.882635+0000","last_clean":"2023-10-11T11:19:01.882635+0000","last_became_active":"2023-10-11T11:15:33.659619+0000","last_became_peered":"2023-10-11T11:15:33.659619+0000","last_unstale":"2023-10-11T11:19:01.882635+0000","last_undegraded":"2023-10-11T11:19:01.882635+0000","last_fullsized":"2023-10-11T11:19:01.882635+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T17:21:16.866392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":517,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-10-11T11:19:02.775128+0000","last_change":"2023-10-11T11:15:31.670813+0000","last_active":"2023-10-11T11:19:02.775128+0000","last_peered":"2023-10-11T11:19:02.775128+0000","last_clean":"2023-10-11T11:15:27.638219+0000","last_became_active":"2023-10-11T11:15:31.670813+0000","last_became_peered":"2023-10-11T11:15:31.670813+0000","last_unstale":"2023-10-11T11:19:02.775128+0000","last_undegraded":"2023-10-11T11:19:02.775128+0000","last_fullsized":"2023-10-11T11:15:31.658785+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:41:50.872346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":516,"reported_epoch":499,"state":"active+undersized","last_fresh":"2023-10-11T11:19:00.631668+0000","last_change":"2023-10-11T11:15:31.666378+0000","last_active":"2023-10-11T11:19:00.631668+0000","last_peered":"2023-10-11T11:19:00.631668+0000","last_clean":"2023-10-11T11:15:27.638303+0000","last_became_active":"2023-10-11T11:15:31.666378+0000","last_became_peered":"2023-10-11T11:15:31.666378+0000","last_unstale":"2023-10-11T11:19:00.631668+0000","last_undegraded":"2023-10-11T11:19:00.631668+0000","last_fullsized":"2023-10-11T11:15:31.649186+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T15:35:19.331209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":516,"reported_epoch":499,"state":"active+undersized","last_fresh":"2023-10-11T11:19:00.636960+0000","last_change":"2023-10-11T11:15:31.679465+0000","last_active":"2023-10-11T11:19:00.636960+0000","last_peered":"2023-10-11T11:19:00.636960+0000","last_clean":"2023-10-11T11:15:27.638749+0000","last_became_active":"2023-10-11T11:15:31.679465+0000","last_became_peered":"2023-10-11T11:15:31.679465+0000","last_unstale":"2023-10-11T11:19:00.636960+0000","last_undegraded":"2023-10-11T11:19:00.636960+0000","last_fullsized":"2023-10-11T11:15:31.659135+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T16:03:42.656692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":11,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738799,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":785348,"kb_used_data":5812,"kb_used_omap":96,"kb_used_meta":779423,"kb_avail":93586492,"statfs":{"total":96636764160,"available":95832567808,"internally_reserved":0,"allocated":5951488,"data_stored":3106918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":98609,"internal_metadata":798129871},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 11:14:37 2023","interfaces":[{"interface":"back","average":{"1min":0.314,"5min":0.316,"15min":0.443},"min":{"1min":0.24399999999999999,"5min":0.14799999999999999,"15min":0.14599999999999999},"max":{"1min":0.48499999999999999,"5min":1.712,"15min":9.6400000000000006},"last":0.31},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.35499999999999998,"15min":0.41299999999999998},"min":{"1min":0.20899999999999999,"5min":0.17199999999999999,"15min":0.16400000000000001},"max":{"1min":3.8180000000000001,"5min":3.8180000000000001,"15min":6.3200000000000003},"last":0.26500000000000001}]},{"osd":1,"last update":"Wed Oct 11 11:18:39 2023","interfaces":[{"interface":"back","average":{"1min":0.27500000000000002,"5min":0.29299999999999998,"15min":0.32400000000000001},"min":{"1min":0.154,"5min":0.154,"15min":0.154},"max":{"1min":0.41999999999999998,"5min":0.59299999999999997,"15min":2.9540000000000002},"last":0.39500000000000002},{"interface":"front","average":{"1min":0.307,"5min":0.35899999999999999,"15min":0.36499999999999999},"min":{"1min":0.18099999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.45100000000000001,"5min":3.2719999999999998,"15min":3.2719999999999998},"last":0.42599999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738756,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1484496,"kb_used_data":406792,"kb_used_omap":107,"kb_used_meta":1077588,"kb_avail":92887344,"statfs":{"total":96636764160,"available":95116640256,"internally_reserved":0,"allocated":416555008,"data_stored":413648921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110309,"internal_metadata":1103450395},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Oct 11 11:14:30 2023","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.35399999999999998,"15min":0.40600000000000003},"min":{"1min":0.18099999999999999,"5min":0.155,"15min":0.14899999999999999},"max":{"1min":1.4319999999999999,"5min":2.3460000000000001,"15min":6.593},"last":0.26200000000000001},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.39100000000000001,"15min":0.48199999999999998},"min":{"1min":0.189,"5min":0.189,"15min":0.183},"max":{"1min":2.2120000000000002,"5min":2.4089999999999998,"15min":6.5960000000000001},"last":0.316}]},{"osd":2,"last update":"Wed Oct 11 11:14:30 2023","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.35499999999999998,"15min":0.47099999999999997},"min":{"1min":0.20300000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":1.2490000000000001,"5min":2.665,"15min":8.4730000000000008},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.375,"15min":0.442},"min":{"1min":0.23200000000000001,"5min":0.20499999999999999,"15min":0.19600000000000001},"max":{"1min":1.28,"5min":2.6230000000000002,"15min":7.0129999999999999},"last":0.33600000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738800,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":826440,"kb_used_data":6392,"kb_used_omap":88,"kb_used_meta":819943,"kb_avail":93545400,"statfs":{"total":96636764160,"available":95790489600,"internally_reserved":0,"allocated":6545408,"data_stored":3697286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":90523,"internal_metadata":839622245},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 11:14:33 2023","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.42799999999999999,"15min":0.41099999999999998},"min":{"1min":0.151,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":1.2809999999999999,"5min":3.5739999999999998,"15min":8.2479999999999993},"last":0.223},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.48299999999999998,"15min":0.49199999999999999},"min":{"1min":0.20699999999999999,"5min":0.182,"15min":0.17999999999999999},"max":{"1min":1.323,"5min":3.6080000000000001,"15min":8.2870000000000008},"last":0.28399999999999997}]},{"osd":2,"last update":"Wed Oct 11 11:18:40 2023","interfaces":[{"interface":"back","average":{"1min":0.33400000000000002,"5min":0.378,"15min":0.35599999999999998},"min":{"1min":0.16500000000000001,"5min":0.14699999999999999,"15min":0.14699999999999999},"max":{"1min":1.244,"5min":7.4829999999999997,"15min":10.617000000000001},"last":0.32000000000000001},{"interface":"front","average":{"1min":0.34599999999999997,"5min":0.39000000000000001,"15min":0.34999999999999998},"min":{"1min":0.20399999999999999,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.81100000000000005,"5min":7.5350000000000001,"15min":7.5350000000000001},"last":0.36199999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-11T11:19:19.120 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~227s 2023-10-11T11:19:21.899 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-10-11T11:19:22.230 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T11:19:22.232 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2023-10-11T11:19:22.244 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":1571,"stamp":"2023-10-11T11:19:20.648544+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":11,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":107,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3096284,"kb_used_data":418996,"kb_used_omap":292,"kb_used_meta":2676955,"kb_avail":280019236,"statfs":{"total":289910292480,"available":286739697664,"internally_reserved":0,"allocated":429051904,"data_stored":420453125,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299441,"internal_metadata":2741202511},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003029"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":517,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-10-11T11:19:02.772828+0000","last_change":"2023-10-11T11:15:31.661747+0000","last_active":"2023-10-11T11:19:02.772828+0000","last_peered":"2023-10-11T11:19:02.772828+0000","last_clean":"2023-10-11T11:15:27.638215+0000","last_became_active":"2023-10-11T11:15:31.661747+0000","last_became_peered":"2023-10-11T11:15:31.661747+0000","last_unstale":"2023-10-11T11:19:02.772828+0000","last_undegraded":"2023-10-11T11:19:02.772828+0000","last_fullsized":"2023-10-11T11:15:31.649336+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:15:25.047039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":28,"reported_epoch":499,"state":"active+undersized","last_fresh":"2023-10-11T11:19:00.628569+0000","last_change":"2023-10-11T11:15:31.671363+0000","last_active":"2023-10-11T11:19:00.628569+0000","last_peered":"2023-10-11T11:19:00.628569+0000","last_clean":"2023-10-11T10:43:40.751300+0000","last_became_active":"2023-10-11T11:15:31.671363+0000","last_became_peered":"2023-10-11T11:15:31.671363+0000","last_unstale":"2023-10-11T11:19:00.628569+0000","last_undegraded":"2023-10-11T11:19:00.628569+0000","last_fullsized":"2023-10-11T11:15:31.652001+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:01:53.884964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":512,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-10-11T11:19:03.429640+0000","last_change":"2023-10-11T10:43:41.772781+0000","last_active":"2023-10-11T11:19:03.429640+0000","last_peered":"2023-10-11T11:19:03.429640+0000","last_clean":"2023-10-11T11:19:03.429640+0000","last_became_active":"2023-10-11T10:43:41.772443+0000","last_became_peered":"2023-10-11T10:43:41.772443+0000","last_unstale":"2023-10-11T11:19:03.429640+0000","last_undegraded":"2023-10-11T11:19:03.429640+0000","last_fullsized":"2023-10-11T11:19:03.429640+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T14:15:30.547284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":513,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-10-11T11:19:02.772211+0000","last_change":"2023-10-11T10:43:41.772615+0000","last_active":"2023-10-11T11:19:02.772211+0000","last_peered":"2023-10-11T11:19:02.772211+0000","last_clean":"2023-10-11T11:19:02.772211+0000","last_became_active":"2023-10-11T10:43:41.772290+0000","last_became_peered":"2023-10-11T10:43:41.772290+0000","last_unstale":"2023-10-11T11:19:02.772211+0000","last_undegraded":"2023-10-11T11:19:02.772211+0000","last_fullsized":"2023-10-11T11:19:02.772211+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:26:06.909963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":647,"reported_epoch":499,"state":"active+undersized+degraded","last_fresh":"2023-10-11T11:19:00.625113+0000","last_change":"2023-10-11T11:15:31.654022+0000","last_active":"2023-10-11T11:19:00.625113+0000","last_peered":"2023-10-11T11:19:00.625113+0000","last_clean":"2023-10-11T11:15:27.636884+0000","last_became_active":"2023-10-11T11:15:31.654022+0000","last_became_peered":"2023-10-11T11:15:31.654022+0000","last_unstale":"2023-10-11T11:19:00.625113+0000","last_undegraded":"2023-10-11T11:15:31.646946+0000","last_fullsized":"2023-10-11T11:15:31.646751+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:36.739637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:36.739637+0000","last_clean_scrub_stamp":"2023-10-11T10:43:36.739637+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T11:33:33.696447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":519,"reported_epoch":499,"state":"active+clean+remapped","last_fresh":"2023-10-11T11:19:01.882635+0000","last_change":"2023-10-11T11:15:33.660200+0000","last_active":"2023-10-11T11:19:01.882635+0000","last_peered":"2023-10-11T11:19:01.882635+0000","last_clean":"2023-10-11T11:19:01.882635+0000","last_became_active":"2023-10-11T11:15:33.659619+0000","last_became_peered":"2023-10-11T11:15:33.659619+0000","last_unstale":"2023-10-11T11:19:01.882635+0000","last_undegraded":"2023-10-11T11:19:01.882635+0000","last_fullsized":"2023-10-11T11:19:01.882635+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T17:21:16.866392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":517,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-10-11T11:19:02.775128+0000","last_change":"2023-10-11T11:15:31.670813+0000","last_active":"2023-10-11T11:19:02.775128+0000","last_peered":"2023-10-11T11:19:02.775128+0000","last_clean":"2023-10-11T11:15:27.638219+0000","last_became_active":"2023-10-11T11:15:31.670813+0000","last_became_peered":"2023-10-11T11:15:31.670813+0000","last_unstale":"2023-10-11T11:19:02.775128+0000","last_undegraded":"2023-10-11T11:19:02.775128+0000","last_fullsized":"2023-10-11T11:15:31.658785+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:41:50.872346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":516,"reported_epoch":499,"state":"active+undersized","last_fresh":"2023-10-11T11:19:00.631668+0000","last_change":"2023-10-11T11:15:31.666378+0000","last_active":"2023-10-11T11:19:00.631668+0000","last_peered":"2023-10-11T11:19:00.631668+0000","last_clean":"2023-10-11T11:15:27.638303+0000","last_became_active":"2023-10-11T11:15:31.666378+0000","last_became_peered":"2023-10-11T11:15:31.666378+0000","last_unstale":"2023-10-11T11:19:00.631668+0000","last_undegraded":"2023-10-11T11:19:00.631668+0000","last_fullsized":"2023-10-11T11:15:31.649186+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T15:35:19.331209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":516,"reported_epoch":499,"state":"active+undersized","last_fresh":"2023-10-11T11:19:00.636960+0000","last_change":"2023-10-11T11:15:31.679465+0000","last_active":"2023-10-11T11:19:00.636960+0000","last_peered":"2023-10-11T11:19:00.636960+0000","last_clean":"2023-10-11T11:15:27.638749+0000","last_became_active":"2023-10-11T11:15:31.679465+0000","last_became_peered":"2023-10-11T11:15:31.679465+0000","last_unstale":"2023-10-11T11:19:00.636960+0000","last_undegraded":"2023-10-11T11:19:00.636960+0000","last_fullsized":"2023-10-11T11:15:31.659135+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T16:03:42.656692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":11,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738800,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":785348,"kb_used_data":5812,"kb_used_omap":96,"kb_used_meta":779423,"kb_avail":93586492,"statfs":{"total":96636764160,"available":95832567808,"internally_reserved":0,"allocated":5951488,"data_stored":3106918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":98609,"internal_metadata":798129871},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 11:14:37 2023","interfaces":[{"interface":"back","average":{"1min":0.314,"5min":0.316,"15min":0.443},"min":{"1min":0.24399999999999999,"5min":0.14799999999999999,"15min":0.14599999999999999},"max":{"1min":0.48499999999999999,"5min":1.712,"15min":9.6400000000000006},"last":0.31},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.35499999999999998,"15min":0.41299999999999998},"min":{"1min":0.20899999999999999,"5min":0.17199999999999999,"15min":0.16400000000000001},"max":{"1min":3.8180000000000001,"5min":3.8180000000000001,"15min":6.3200000000000003},"last":0.26500000000000001}]},{"osd":1,"last update":"Wed Oct 11 11:18:39 2023","interfaces":[{"interface":"back","average":{"1min":0.27500000000000002,"5min":0.29299999999999998,"15min":0.32400000000000001},"min":{"1min":0.154,"5min":0.154,"15min":0.154},"max":{"1min":0.41999999999999998,"5min":0.59299999999999997,"15min":2.9540000000000002},"last":0.26400000000000001},{"interface":"front","average":{"1min":0.307,"5min":0.35899999999999999,"15min":0.36499999999999999},"min":{"1min":0.18099999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.45100000000000001,"5min":3.2719999999999998,"15min":3.2719999999999998},"last":0.308}]}]},{"osd":0,"up_from":8,"seq":34359738756,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1484496,"kb_used_data":406792,"kb_used_omap":107,"kb_used_meta":1077588,"kb_avail":92887344,"statfs":{"total":96636764160,"available":95116640256,"internally_reserved":0,"allocated":416555008,"data_stored":413648921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110309,"internal_metadata":1103450395},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Oct 11 11:14:30 2023","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.35399999999999998,"15min":0.40600000000000003},"min":{"1min":0.18099999999999999,"5min":0.155,"15min":0.14899999999999999},"max":{"1min":1.4319999999999999,"5min":2.3460000000000001,"15min":6.593},"last":0.26200000000000001},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.39100000000000001,"15min":0.48199999999999998},"min":{"1min":0.189,"5min":0.189,"15min":0.183},"max":{"1min":2.2120000000000002,"5min":2.4089999999999998,"15min":6.5960000000000001},"last":0.316}]},{"osd":2,"last update":"Wed Oct 11 11:14:30 2023","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.35499999999999998,"15min":0.47099999999999997},"min":{"1min":0.20300000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":1.2490000000000001,"5min":2.665,"15min":8.4730000000000008},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.375,"15min":0.442},"min":{"1min":0.23200000000000001,"5min":0.20499999999999999,"15min":0.19600000000000001},"max":{"1min":1.28,"5min":2.6230000000000002,"15min":7.0129999999999999},"last":0.33600000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738801,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":826440,"kb_used_data":6392,"kb_used_omap":88,"kb_used_meta":819943,"kb_avail":93545400,"statfs":{"total":96636764160,"available":95790489600,"internally_reserved":0,"allocated":6545408,"data_stored":3697286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":90523,"internal_metadata":839622245},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 11:14:33 2023","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.42799999999999999,"15min":0.41099999999999998},"min":{"1min":0.151,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":1.2809999999999999,"5min":3.5739999999999998,"15min":8.2479999999999993},"last":0.223},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.48299999999999998,"15min":0.49199999999999999},"min":{"1min":0.20699999999999999,"5min":0.182,"15min":0.17999999999999999},"max":{"1min":1.323,"5min":3.6080000000000001,"15min":8.2870000000000008},"last":0.28399999999999997}]},{"osd":2,"last update":"Wed Oct 11 11:18:40 2023","interfaces":[{"interface":"back","average":{"1min":0.33400000000000002,"5min":0.378,"15min":0.35599999999999998},"min":{"1min":0.16500000000000001,"5min":0.14699999999999999,"15min":0.14699999999999999},"max":{"1min":1.244,"5min":7.4829999999999997,"15min":10.617000000000001},"last":0.27600000000000002},{"interface":"front","average":{"1min":0.34599999999999997,"5min":0.39000000000000001,"15min":0.34999999999999998},"min":{"1min":0.20399999999999999,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.81100000000000005,"5min":7.5350000000000001,"15min":7.5350000000000001},"last":0.315}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-11T11:19:22.245 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-10-11T11:19:22.562 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T11:19:22.564 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2023-10-11T11:19:22.576 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":1571,"stamp":"2023-10-11T11:19:20.648544+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":11,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":107,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3096284,"kb_used_data":418996,"kb_used_omap":292,"kb_used_meta":2676955,"kb_avail":280019236,"statfs":{"total":289910292480,"available":286739697664,"internally_reserved":0,"allocated":429051904,"data_stored":420453125,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299441,"internal_metadata":2741202511},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003029"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":517,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-10-11T11:19:02.772828+0000","last_change":"2023-10-11T11:15:31.661747+0000","last_active":"2023-10-11T11:19:02.772828+0000","last_peered":"2023-10-11T11:19:02.772828+0000","last_clean":"2023-10-11T11:15:27.638215+0000","last_became_active":"2023-10-11T11:15:31.661747+0000","last_became_peered":"2023-10-11T11:15:31.661747+0000","last_unstale":"2023-10-11T11:19:02.772828+0000","last_undegraded":"2023-10-11T11:19:02.772828+0000","last_fullsized":"2023-10-11T11:15:31.649336+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:15:25.047039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":28,"reported_epoch":499,"state":"active+undersized","last_fresh":"2023-10-11T11:19:00.628569+0000","last_change":"2023-10-11T11:15:31.671363+0000","last_active":"2023-10-11T11:19:00.628569+0000","last_peered":"2023-10-11T11:19:00.628569+0000","last_clean":"2023-10-11T10:43:40.751300+0000","last_became_active":"2023-10-11T11:15:31.671363+0000","last_became_peered":"2023-10-11T11:15:31.671363+0000","last_unstale":"2023-10-11T11:19:00.628569+0000","last_undegraded":"2023-10-11T11:19:00.628569+0000","last_fullsized":"2023-10-11T11:15:31.652001+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:01:53.884964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":512,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-10-11T11:19:03.429640+0000","last_change":"2023-10-11T10:43:41.772781+0000","last_active":"2023-10-11T11:19:03.429640+0000","last_peered":"2023-10-11T11:19:03.429640+0000","last_clean":"2023-10-11T11:19:03.429640+0000","last_became_active":"2023-10-11T10:43:41.772443+0000","last_became_peered":"2023-10-11T10:43:41.772443+0000","last_unstale":"2023-10-11T11:19:03.429640+0000","last_undegraded":"2023-10-11T11:19:03.429640+0000","last_fullsized":"2023-10-11T11:19:03.429640+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T14:15:30.547284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":513,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-10-11T11:19:02.772211+0000","last_change":"2023-10-11T10:43:41.772615+0000","last_active":"2023-10-11T11:19:02.772211+0000","last_peered":"2023-10-11T11:19:02.772211+0000","last_clean":"2023-10-11T11:19:02.772211+0000","last_became_active":"2023-10-11T10:43:41.772290+0000","last_became_peered":"2023-10-11T10:43:41.772290+0000","last_unstale":"2023-10-11T11:19:02.772211+0000","last_undegraded":"2023-10-11T11:19:02.772211+0000","last_fullsized":"2023-10-11T11:19:02.772211+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:26:06.909963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":647,"reported_epoch":499,"state":"active+undersized+degraded","last_fresh":"2023-10-11T11:19:00.625113+0000","last_change":"2023-10-11T11:15:31.654022+0000","last_active":"2023-10-11T11:19:00.625113+0000","last_peered":"2023-10-11T11:19:00.625113+0000","last_clean":"2023-10-11T11:15:27.636884+0000","last_became_active":"2023-10-11T11:15:31.654022+0000","last_became_peered":"2023-10-11T11:15:31.654022+0000","last_unstale":"2023-10-11T11:19:00.625113+0000","last_undegraded":"2023-10-11T11:15:31.646946+0000","last_fullsized":"2023-10-11T11:15:31.646751+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:36.739637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:36.739637+0000","last_clean_scrub_stamp":"2023-10-11T10:43:36.739637+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T11:33:33.696447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":519,"reported_epoch":499,"state":"active+clean+remapped","last_fresh":"2023-10-11T11:19:01.882635+0000","last_change":"2023-10-11T11:15:33.660200+0000","last_active":"2023-10-11T11:19:01.882635+0000","last_peered":"2023-10-11T11:19:01.882635+0000","last_clean":"2023-10-11T11:19:01.882635+0000","last_became_active":"2023-10-11T11:15:33.659619+0000","last_became_peered":"2023-10-11T11:15:33.659619+0000","last_unstale":"2023-10-11T11:19:01.882635+0000","last_undegraded":"2023-10-11T11:19:01.882635+0000","last_fullsized":"2023-10-11T11:19:01.882635+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T17:21:16.866392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":517,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-10-11T11:19:02.775128+0000","last_change":"2023-10-11T11:15:31.670813+0000","last_active":"2023-10-11T11:19:02.775128+0000","last_peered":"2023-10-11T11:19:02.775128+0000","last_clean":"2023-10-11T11:15:27.638219+0000","last_became_active":"2023-10-11T11:15:31.670813+0000","last_became_peered":"2023-10-11T11:15:31.670813+0000","last_unstale":"2023-10-11T11:19:02.775128+0000","last_undegraded":"2023-10-11T11:19:02.775128+0000","last_fullsized":"2023-10-11T11:15:31.658785+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:41:50.872346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":516,"reported_epoch":499,"state":"active+undersized","last_fresh":"2023-10-11T11:19:00.631668+0000","last_change":"2023-10-11T11:15:31.666378+0000","last_active":"2023-10-11T11:19:00.631668+0000","last_peered":"2023-10-11T11:19:00.631668+0000","last_clean":"2023-10-11T11:15:27.638303+0000","last_became_active":"2023-10-11T11:15:31.666378+0000","last_became_peered":"2023-10-11T11:15:31.666378+0000","last_unstale":"2023-10-11T11:19:00.631668+0000","last_undegraded":"2023-10-11T11:19:00.631668+0000","last_fullsized":"2023-10-11T11:15:31.649186+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T15:35:19.331209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":516,"reported_epoch":499,"state":"active+undersized","last_fresh":"2023-10-11T11:19:00.636960+0000","last_change":"2023-10-11T11:15:31.679465+0000","last_active":"2023-10-11T11:19:00.636960+0000","last_peered":"2023-10-11T11:19:00.636960+0000","last_clean":"2023-10-11T11:15:27.638749+0000","last_became_active":"2023-10-11T11:15:31.679465+0000","last_became_peered":"2023-10-11T11:15:31.679465+0000","last_unstale":"2023-10-11T11:19:00.636960+0000","last_undegraded":"2023-10-11T11:19:00.636960+0000","last_fullsized":"2023-10-11T11:15:31.659135+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T16:03:42.656692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":11,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738800,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":785348,"kb_used_data":5812,"kb_used_omap":96,"kb_used_meta":779423,"kb_avail":93586492,"statfs":{"total":96636764160,"available":95832567808,"internally_reserved":0,"allocated":5951488,"data_stored":3106918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":98609,"internal_metadata":798129871},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 11:14:37 2023","interfaces":[{"interface":"back","average":{"1min":0.314,"5min":0.316,"15min":0.443},"min":{"1min":0.24399999999999999,"5min":0.14799999999999999,"15min":0.14599999999999999},"max":{"1min":0.48499999999999999,"5min":1.712,"15min":9.6400000000000006},"last":0.31},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.35499999999999998,"15min":0.41299999999999998},"min":{"1min":0.20899999999999999,"5min":0.17199999999999999,"15min":0.16400000000000001},"max":{"1min":3.8180000000000001,"5min":3.8180000000000001,"15min":6.3200000000000003},"last":0.26500000000000001}]},{"osd":1,"last update":"Wed Oct 11 11:18:39 2023","interfaces":[{"interface":"back","average":{"1min":0.27500000000000002,"5min":0.29299999999999998,"15min":0.32400000000000001},"min":{"1min":0.154,"5min":0.154,"15min":0.154},"max":{"1min":0.41999999999999998,"5min":0.59299999999999997,"15min":2.9540000000000002},"last":0.26400000000000001},{"interface":"front","average":{"1min":0.307,"5min":0.35899999999999999,"15min":0.36499999999999999},"min":{"1min":0.18099999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.45100000000000001,"5min":3.2719999999999998,"15min":3.2719999999999998},"last":0.308}]}]},{"osd":0,"up_from":8,"seq":34359738756,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1484496,"kb_used_data":406792,"kb_used_omap":107,"kb_used_meta":1077588,"kb_avail":92887344,"statfs":{"total":96636764160,"available":95116640256,"internally_reserved":0,"allocated":416555008,"data_stored":413648921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110309,"internal_metadata":1103450395},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Oct 11 11:14:30 2023","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.35399999999999998,"15min":0.40600000000000003},"min":{"1min":0.18099999999999999,"5min":0.155,"15min":0.14899999999999999},"max":{"1min":1.4319999999999999,"5min":2.3460000000000001,"15min":6.593},"last":0.26200000000000001},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.39100000000000001,"15min":0.48199999999999998},"min":{"1min":0.189,"5min":0.189,"15min":0.183},"max":{"1min":2.2120000000000002,"5min":2.4089999999999998,"15min":6.5960000000000001},"last":0.316}]},{"osd":2,"last update":"Wed Oct 11 11:14:30 2023","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.35499999999999998,"15min":0.47099999999999997},"min":{"1min":0.20300000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":1.2490000000000001,"5min":2.665,"15min":8.4730000000000008},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.375,"15min":0.442},"min":{"1min":0.23200000000000001,"5min":0.20499999999999999,"15min":0.19600000000000001},"max":{"1min":1.28,"5min":2.6230000000000002,"15min":7.0129999999999999},"last":0.33600000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738801,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":826440,"kb_used_data":6392,"kb_used_omap":88,"kb_used_meta":819943,"kb_avail":93545400,"statfs":{"total":96636764160,"available":95790489600,"internally_reserved":0,"allocated":6545408,"data_stored":3697286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":90523,"internal_metadata":839622245},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 11:14:33 2023","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.42799999999999999,"15min":0.41099999999999998},"min":{"1min":0.151,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":1.2809999999999999,"5min":3.5739999999999998,"15min":8.2479999999999993},"last":0.223},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.48299999999999998,"15min":0.49199999999999999},"min":{"1min":0.20699999999999999,"5min":0.182,"15min":0.17999999999999999},"max":{"1min":1.323,"5min":3.6080000000000001,"15min":8.2870000000000008},"last":0.28399999999999997}]},{"osd":2,"last update":"Wed Oct 11 11:18:40 2023","interfaces":[{"interface":"back","average":{"1min":0.33400000000000002,"5min":0.378,"15min":0.35599999999999998},"min":{"1min":0.16500000000000001,"5min":0.14699999999999999,"15min":0.14699999999999999},"max":{"1min":1.244,"5min":7.4829999999999997,"15min":10.617000000000001},"last":0.27600000000000002},{"interface":"front","average":{"1min":0.34599999999999997,"5min":0.39000000000000001,"15min":0.34999999999999998},"min":{"1min":0.20399999999999999,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.81100000000000005,"5min":7.5350000000000001,"15min":7.5350000000000001},"last":0.315}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-11T11:19:22.577 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-10-11T11:19:22.577 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 517, 'reported_epoch': 500, 'state': 'active+undersized', 'last_fresh': '2023-10-11T11:19:02.772828+0000', 'last_change': '2023-10-11T11:15:31.661747+0000', 'last_active': '2023-10-11T11:19:02.772828+0000', 'last_peered': '2023-10-11T11:19:02.772828+0000', 'last_clean': '2023-10-11T11:15:27.638215+0000', 'last_became_active': '2023-10-11T11:15:31.661747+0000', 'last_became_peered': '2023-10-11T11:15:31.661747+0000', 'last_unstale': '2023-10-11T11:19:02.772828+0000', 'last_undegraded': '2023-10-11T11:19:02.772828+0000', 'last_fullsized': '2023-10-11T11:15:31.649336+0000', 'mapping_epoch': 480, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_clean_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-12T22:15:25.047039+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-10-11T11:19:22.578 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-10-11T11:19:22.578 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 28, 'reported_epoch': 499, 'state': 'active+undersized', 'last_fresh': '2023-10-11T11:19:00.628569+0000', 'last_change': '2023-10-11T11:15:31.671363+0000', 'last_active': '2023-10-11T11:19:00.628569+0000', 'last_peered': '2023-10-11T11:19:00.628569+0000', 'last_clean': '2023-10-11T10:43:40.751300+0000', 'last_became_active': '2023-10-11T11:15:31.671363+0000', 'last_became_peered': '2023-10-11T11:15:31.671363+0000', 'last_unstale': '2023-10-11T11:19:00.628569+0000', 'last_undegraded': '2023-10-11T11:19:00.628569+0000', 'last_fullsized': '2023-10-11T11:15:31.652001+0000', 'mapping_epoch': 480, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_clean_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-12T21:01:53.884964+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-10-11T11:19:22.578 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-10-11T11:19:22.579 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "11'97", 'reported_seq': 647, 'reported_epoch': 499, 'state': 'active+undersized+degraded', 'last_fresh': '2023-10-11T11:19:00.625113+0000', 'last_change': '2023-10-11T11:15:31.654022+0000', 'last_active': '2023-10-11T11:19:00.625113+0000', 'last_peered': '2023-10-11T11:19:00.625113+0000', 'last_clean': '2023-10-11T11:15:27.636884+0000', 'last_became_active': '2023-10-11T11:15:31.654022+0000', 'last_became_peered': '2023-10-11T11:15:31.654022+0000', 'last_unstale': '2023-10-11T11:19:00.625113+0000', 'last_undegraded': '2023-10-11T11:15:31.646946+0000', 'last_fullsized': '2023-10-11T11:15:31.646751+0000', 'mapping_epoch': 480, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-11T10:43:36.739637+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-11T10:43:36.739637+0000', 'last_clean_scrub_stamp': '2023-10-11T10:43:36.739637+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-12T11:33:33.696447+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-10-11T11:19:22.579 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.0 is not active+clean 2023-10-11T11:19:22.579 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 519, 'reported_epoch': 499, 'state': 'active+clean+remapped', 'last_fresh': '2023-10-11T11:19:01.882635+0000', 'last_change': '2023-10-11T11:15:33.660200+0000', 'last_active': '2023-10-11T11:19:01.882635+0000', 'last_peered': '2023-10-11T11:19:01.882635+0000', 'last_clean': '2023-10-11T11:19:01.882635+0000', 'last_became_active': '2023-10-11T11:15:33.659619+0000', 'last_became_peered': '2023-10-11T11:15:33.659619+0000', 'last_unstale': '2023-10-11T11:19:01.882635+0000', 'last_undegraded': '2023-10-11T11:19:01.882635+0000', 'last_fullsized': '2023-10-11T11:19:01.882635+0000', 'mapping_epoch': 482, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 483, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_clean_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-12T17:21:16.866392+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2, 1], 'avail_no_missing': ['2', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-10-11T11:19:22.579 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-10-11T11:19:22.580 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 517, 'reported_epoch': 500, 'state': 'active+undersized', 'last_fresh': '2023-10-11T11:19:02.775128+0000', 'last_change': '2023-10-11T11:15:31.670813+0000', 'last_active': '2023-10-11T11:19:02.775128+0000', 'last_peered': '2023-10-11T11:19:02.775128+0000', 'last_clean': '2023-10-11T11:15:27.638219+0000', 'last_became_active': '2023-10-11T11:15:31.670813+0000', 'last_became_peered': '2023-10-11T11:15:31.670813+0000', 'last_unstale': '2023-10-11T11:19:02.775128+0000', 'last_undegraded': '2023-10-11T11:19:02.775128+0000', 'last_fullsized': '2023-10-11T11:15:31.658785+0000', 'mapping_epoch': 480, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_clean_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-12T21:41:50.872346+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-10-11T11:19:22.580 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-10-11T11:19:22.580 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 516, 'reported_epoch': 499, 'state': 'active+undersized', 'last_fresh': '2023-10-11T11:19:00.631668+0000', 'last_change': '2023-10-11T11:15:31.666378+0000', 'last_active': '2023-10-11T11:19:00.631668+0000', 'last_peered': '2023-10-11T11:19:00.631668+0000', 'last_clean': '2023-10-11T11:15:27.638303+0000', 'last_became_active': '2023-10-11T11:15:31.666378+0000', 'last_became_peered': '2023-10-11T11:15:31.666378+0000', 'last_unstale': '2023-10-11T11:19:00.631668+0000', 'last_undegraded': '2023-10-11T11:19:00.631668+0000', 'last_fullsized': '2023-10-11T11:15:31.649186+0000', 'mapping_epoch': 480, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_clean_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-12T15:35:19.331209+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-10-11T11:19:22.581 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-10-11T11:19:22.581 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 516, 'reported_epoch': 499, 'state': 'active+undersized', 'last_fresh': '2023-10-11T11:19:00.636960+0000', 'last_change': '2023-10-11T11:15:31.679465+0000', 'last_active': '2023-10-11T11:19:00.636960+0000', 'last_peered': '2023-10-11T11:19:00.636960+0000', 'last_clean': '2023-10-11T11:15:27.638749+0000', 'last_became_active': '2023-10-11T11:15:31.679465+0000', 'last_became_peered': '2023-10-11T11:15:31.679465+0000', 'last_unstale': '2023-10-11T11:19:00.636960+0000', 'last_undegraded': '2023-10-11T11:19:00.636960+0000', 'last_fullsized': '2023-10-11T11:15:31.659135+0000', 'mapping_epoch': 480, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_clean_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-12T16:03:42.656692+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-10-11T11:19:22.581 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-10-11T11:19:22.975 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T11:19:22.976 INFO:teuthology.orchestra.run.smithi111.stdout:{"fsid":"022ce6f2-9a5b-45ad-8b77-a1332d92f228","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2158,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":500,"num_osds":3,"num_up_osds":2,"osd_up_since":1697022930,"num_in_osds":3,"osd_in_since":1697021005,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":2},{"state_name":"active+undersized+degraded","count":1},{"state_name":"active+clean+remapped","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3170594816,"bytes_avail":286739697664,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":36,"modified":"2023-10-11T11:19:04.644408+0000","services":{}},"progress_events":{}} 2023-10-11T11:19:22.989 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-10-11T11:19:22.990 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-10-11T11:19:23.307 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T11:19:23.309 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2023-10-11T11:19:23.321 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":1572,"stamp":"2023-10-11T11:19:22.648971+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":11,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":107,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3096284,"kb_used_data":418996,"kb_used_omap":292,"kb_used_meta":2676955,"kb_avail":280019236,"statfs":{"total":289910292480,"available":286739697664,"internally_reserved":0,"allocated":429051904,"data_stored":420453125,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299441,"internal_metadata":2741202511},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002978"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":517,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-10-11T11:19:02.772828+0000","last_change":"2023-10-11T11:15:31.661747+0000","last_active":"2023-10-11T11:19:02.772828+0000","last_peered":"2023-10-11T11:19:02.772828+0000","last_clean":"2023-10-11T11:15:27.638215+0000","last_became_active":"2023-10-11T11:15:31.661747+0000","last_became_peered":"2023-10-11T11:15:31.661747+0000","last_unstale":"2023-10-11T11:19:02.772828+0000","last_undegraded":"2023-10-11T11:19:02.772828+0000","last_fullsized":"2023-10-11T11:15:31.649336+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:15:25.047039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":28,"reported_epoch":499,"state":"active+undersized","last_fresh":"2023-10-11T11:19:00.628569+0000","last_change":"2023-10-11T11:15:31.671363+0000","last_active":"2023-10-11T11:19:00.628569+0000","last_peered":"2023-10-11T11:19:00.628569+0000","last_clean":"2023-10-11T10:43:40.751300+0000","last_became_active":"2023-10-11T11:15:31.671363+0000","last_became_peered":"2023-10-11T11:15:31.671363+0000","last_unstale":"2023-10-11T11:19:00.628569+0000","last_undegraded":"2023-10-11T11:19:00.628569+0000","last_fullsized":"2023-10-11T11:15:31.652001+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:01:53.884964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":512,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-10-11T11:19:03.429640+0000","last_change":"2023-10-11T10:43:41.772781+0000","last_active":"2023-10-11T11:19:03.429640+0000","last_peered":"2023-10-11T11:19:03.429640+0000","last_clean":"2023-10-11T11:19:03.429640+0000","last_became_active":"2023-10-11T10:43:41.772443+0000","last_became_peered":"2023-10-11T10:43:41.772443+0000","last_unstale":"2023-10-11T11:19:03.429640+0000","last_undegraded":"2023-10-11T11:19:03.429640+0000","last_fullsized":"2023-10-11T11:19:03.429640+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T14:15:30.547284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":513,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-10-11T11:19:02.772211+0000","last_change":"2023-10-11T10:43:41.772615+0000","last_active":"2023-10-11T11:19:02.772211+0000","last_peered":"2023-10-11T11:19:02.772211+0000","last_clean":"2023-10-11T11:19:02.772211+0000","last_became_active":"2023-10-11T10:43:41.772290+0000","last_became_peered":"2023-10-11T10:43:41.772290+0000","last_unstale":"2023-10-11T11:19:02.772211+0000","last_undegraded":"2023-10-11T11:19:02.772211+0000","last_fullsized":"2023-10-11T11:19:02.772211+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:26:06.909963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":647,"reported_epoch":499,"state":"active+undersized+degraded","last_fresh":"2023-10-11T11:19:00.625113+0000","last_change":"2023-10-11T11:15:31.654022+0000","last_active":"2023-10-11T11:19:00.625113+0000","last_peered":"2023-10-11T11:19:00.625113+0000","last_clean":"2023-10-11T11:15:27.636884+0000","last_became_active":"2023-10-11T11:15:31.654022+0000","last_became_peered":"2023-10-11T11:15:31.654022+0000","last_unstale":"2023-10-11T11:19:00.625113+0000","last_undegraded":"2023-10-11T11:15:31.646946+0000","last_fullsized":"2023-10-11T11:15:31.646751+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:36.739637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:36.739637+0000","last_clean_scrub_stamp":"2023-10-11T10:43:36.739637+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T11:33:33.696447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":519,"reported_epoch":499,"state":"active+clean+remapped","last_fresh":"2023-10-11T11:19:01.882635+0000","last_change":"2023-10-11T11:15:33.660200+0000","last_active":"2023-10-11T11:19:01.882635+0000","last_peered":"2023-10-11T11:19:01.882635+0000","last_clean":"2023-10-11T11:19:01.882635+0000","last_became_active":"2023-10-11T11:15:33.659619+0000","last_became_peered":"2023-10-11T11:15:33.659619+0000","last_unstale":"2023-10-11T11:19:01.882635+0000","last_undegraded":"2023-10-11T11:19:01.882635+0000","last_fullsized":"2023-10-11T11:19:01.882635+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T17:21:16.866392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":517,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-10-11T11:19:02.775128+0000","last_change":"2023-10-11T11:15:31.670813+0000","last_active":"2023-10-11T11:19:02.775128+0000","last_peered":"2023-10-11T11:19:02.775128+0000","last_clean":"2023-10-11T11:15:27.638219+0000","last_became_active":"2023-10-11T11:15:31.670813+0000","last_became_peered":"2023-10-11T11:15:31.670813+0000","last_unstale":"2023-10-11T11:19:02.775128+0000","last_undegraded":"2023-10-11T11:19:02.775128+0000","last_fullsized":"2023-10-11T11:15:31.658785+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:41:50.872346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":516,"reported_epoch":499,"state":"active+undersized","last_fresh":"2023-10-11T11:19:00.631668+0000","last_change":"2023-10-11T11:15:31.666378+0000","last_active":"2023-10-11T11:19:00.631668+0000","last_peered":"2023-10-11T11:19:00.631668+0000","last_clean":"2023-10-11T11:15:27.638303+0000","last_became_active":"2023-10-11T11:15:31.666378+0000","last_became_peered":"2023-10-11T11:15:31.666378+0000","last_unstale":"2023-10-11T11:19:00.631668+0000","last_undegraded":"2023-10-11T11:19:00.631668+0000","last_fullsized":"2023-10-11T11:15:31.649186+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T15:35:19.331209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":516,"reported_epoch":499,"state":"active+undersized","last_fresh":"2023-10-11T11:19:00.636960+0000","last_change":"2023-10-11T11:15:31.679465+0000","last_active":"2023-10-11T11:19:00.636960+0000","last_peered":"2023-10-11T11:19:00.636960+0000","last_clean":"2023-10-11T11:15:27.638749+0000","last_became_active":"2023-10-11T11:15:31.679465+0000","last_became_peered":"2023-10-11T11:15:31.679465+0000","last_unstale":"2023-10-11T11:19:00.636960+0000","last_undegraded":"2023-10-11T11:19:00.636960+0000","last_fullsized":"2023-10-11T11:15:31.659135+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T16:03:42.656692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":11,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738800,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":785348,"kb_used_data":5812,"kb_used_omap":96,"kb_used_meta":779423,"kb_avail":93586492,"statfs":{"total":96636764160,"available":95832567808,"internally_reserved":0,"allocated":5951488,"data_stored":3106918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":98609,"internal_metadata":798129871},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 11:14:37 2023","interfaces":[{"interface":"back","average":{"1min":0.314,"5min":0.316,"15min":0.443},"min":{"1min":0.24399999999999999,"5min":0.14799999999999999,"15min":0.14599999999999999},"max":{"1min":0.48499999999999999,"5min":1.712,"15min":9.6400000000000006},"last":0.31},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.35499999999999998,"15min":0.41299999999999998},"min":{"1min":0.20899999999999999,"5min":0.17199999999999999,"15min":0.16400000000000001},"max":{"1min":3.8180000000000001,"5min":3.8180000000000001,"15min":6.3200000000000003},"last":0.26500000000000001}]},{"osd":1,"last update":"Wed Oct 11 11:18:39 2023","interfaces":[{"interface":"back","average":{"1min":0.27500000000000002,"5min":0.29299999999999998,"15min":0.32400000000000001},"min":{"1min":0.154,"5min":0.154,"15min":0.154},"max":{"1min":0.41999999999999998,"5min":0.59299999999999997,"15min":2.9540000000000002},"last":0.26400000000000001},{"interface":"front","average":{"1min":0.307,"5min":0.35899999999999999,"15min":0.36499999999999999},"min":{"1min":0.18099999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.45100000000000001,"5min":3.2719999999999998,"15min":3.2719999999999998},"last":0.308}]}]},{"osd":0,"up_from":8,"seq":34359738756,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1484496,"kb_used_data":406792,"kb_used_omap":107,"kb_used_meta":1077588,"kb_avail":92887344,"statfs":{"total":96636764160,"available":95116640256,"internally_reserved":0,"allocated":416555008,"data_stored":413648921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110309,"internal_metadata":1103450395},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Oct 11 11:14:30 2023","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.35399999999999998,"15min":0.40600000000000003},"min":{"1min":0.18099999999999999,"5min":0.155,"15min":0.14899999999999999},"max":{"1min":1.4319999999999999,"5min":2.3460000000000001,"15min":6.593},"last":0.26200000000000001},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.39100000000000001,"15min":0.48199999999999998},"min":{"1min":0.189,"5min":0.189,"15min":0.183},"max":{"1min":2.2120000000000002,"5min":2.4089999999999998,"15min":6.5960000000000001},"last":0.316}]},{"osd":2,"last update":"Wed Oct 11 11:14:30 2023","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.35499999999999998,"15min":0.47099999999999997},"min":{"1min":0.20300000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":1.2490000000000001,"5min":2.665,"15min":8.4730000000000008},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.375,"15min":0.442},"min":{"1min":0.23200000000000001,"5min":0.20499999999999999,"15min":0.19600000000000001},"max":{"1min":1.28,"5min":2.6230000000000002,"15min":7.0129999999999999},"last":0.33600000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738801,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":826440,"kb_used_data":6392,"kb_used_omap":88,"kb_used_meta":819943,"kb_avail":93545400,"statfs":{"total":96636764160,"available":95790489600,"internally_reserved":0,"allocated":6545408,"data_stored":3697286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":90523,"internal_metadata":839622245},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 11:14:33 2023","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.42799999999999999,"15min":0.41099999999999998},"min":{"1min":0.151,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":1.2809999999999999,"5min":3.5739999999999998,"15min":8.2479999999999993},"last":0.223},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.48299999999999998,"15min":0.49199999999999999},"min":{"1min":0.20699999999999999,"5min":0.182,"15min":0.17999999999999999},"max":{"1min":1.323,"5min":3.6080000000000001,"15min":8.2870000000000008},"last":0.28399999999999997}]},{"osd":2,"last update":"Wed Oct 11 11:18:40 2023","interfaces":[{"interface":"back","average":{"1min":0.33400000000000002,"5min":0.378,"15min":0.35599999999999998},"min":{"1min":0.16500000000000001,"5min":0.14699999999999999,"15min":0.14699999999999999},"max":{"1min":1.244,"5min":7.4829999999999997,"15min":10.617000000000001},"last":0.27600000000000002},{"interface":"front","average":{"1min":0.34599999999999997,"5min":0.39000000000000001,"15min":0.34999999999999998},"min":{"1min":0.20399999999999999,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.81100000000000005,"5min":7.5350000000000001,"15min":7.5350000000000001},"last":0.315}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-11T11:19:24.524 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~232s 2023-10-11T11:19:26.323 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-10-11T11:19:26.652 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T11:19:26.654 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2023-10-11T11:19:26.666 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":1574,"stamp":"2023-10-11T11:19:26.649964+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":11,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":107,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3096284,"kb_used_data":418996,"kb_used_omap":292,"kb_used_meta":2676955,"kb_avail":280019236,"statfs":{"total":289910292480,"available":286739697664,"internally_reserved":0,"allocated":429051904,"data_stored":420453125,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299441,"internal_metadata":2741202511},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002917"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":517,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-10-11T11:19:02.772828+0000","last_change":"2023-10-11T11:15:31.661747+0000","last_active":"2023-10-11T11:19:02.772828+0000","last_peered":"2023-10-11T11:19:02.772828+0000","last_clean":"2023-10-11T11:15:27.638215+0000","last_became_active":"2023-10-11T11:15:31.661747+0000","last_became_peered":"2023-10-11T11:15:31.661747+0000","last_unstale":"2023-10-11T11:19:02.772828+0000","last_undegraded":"2023-10-11T11:19:02.772828+0000","last_fullsized":"2023-10-11T11:15:31.649336+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:15:25.047039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":28,"reported_epoch":499,"state":"active+undersized","last_fresh":"2023-10-11T11:19:00.628569+0000","last_change":"2023-10-11T11:15:31.671363+0000","last_active":"2023-10-11T11:19:00.628569+0000","last_peered":"2023-10-11T11:19:00.628569+0000","last_clean":"2023-10-11T10:43:40.751300+0000","last_became_active":"2023-10-11T11:15:31.671363+0000","last_became_peered":"2023-10-11T11:15:31.671363+0000","last_unstale":"2023-10-11T11:19:00.628569+0000","last_undegraded":"2023-10-11T11:19:00.628569+0000","last_fullsized":"2023-10-11T11:15:31.652001+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:01:53.884964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":512,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-10-11T11:19:03.429640+0000","last_change":"2023-10-11T10:43:41.772781+0000","last_active":"2023-10-11T11:19:03.429640+0000","last_peered":"2023-10-11T11:19:03.429640+0000","last_clean":"2023-10-11T11:19:03.429640+0000","last_became_active":"2023-10-11T10:43:41.772443+0000","last_became_peered":"2023-10-11T10:43:41.772443+0000","last_unstale":"2023-10-11T11:19:03.429640+0000","last_undegraded":"2023-10-11T11:19:03.429640+0000","last_fullsized":"2023-10-11T11:19:03.429640+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T14:15:30.547284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":513,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-10-11T11:19:02.772211+0000","last_change":"2023-10-11T10:43:41.772615+0000","last_active":"2023-10-11T11:19:02.772211+0000","last_peered":"2023-10-11T11:19:02.772211+0000","last_clean":"2023-10-11T11:19:02.772211+0000","last_became_active":"2023-10-11T10:43:41.772290+0000","last_became_peered":"2023-10-11T10:43:41.772290+0000","last_unstale":"2023-10-11T11:19:02.772211+0000","last_undegraded":"2023-10-11T11:19:02.772211+0000","last_fullsized":"2023-10-11T11:19:02.772211+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:26:06.909963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":647,"reported_epoch":499,"state":"active+undersized+degraded","last_fresh":"2023-10-11T11:19:00.625113+0000","last_change":"2023-10-11T11:15:31.654022+0000","last_active":"2023-10-11T11:19:00.625113+0000","last_peered":"2023-10-11T11:19:00.625113+0000","last_clean":"2023-10-11T11:15:27.636884+0000","last_became_active":"2023-10-11T11:15:31.654022+0000","last_became_peered":"2023-10-11T11:15:31.654022+0000","last_unstale":"2023-10-11T11:19:00.625113+0000","last_undegraded":"2023-10-11T11:15:31.646946+0000","last_fullsized":"2023-10-11T11:15:31.646751+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:36.739637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:36.739637+0000","last_clean_scrub_stamp":"2023-10-11T10:43:36.739637+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T11:33:33.696447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":519,"reported_epoch":499,"state":"active+clean+remapped","last_fresh":"2023-10-11T11:19:01.882635+0000","last_change":"2023-10-11T11:15:33.660200+0000","last_active":"2023-10-11T11:19:01.882635+0000","last_peered":"2023-10-11T11:19:01.882635+0000","last_clean":"2023-10-11T11:19:01.882635+0000","last_became_active":"2023-10-11T11:15:33.659619+0000","last_became_peered":"2023-10-11T11:15:33.659619+0000","last_unstale":"2023-10-11T11:19:01.882635+0000","last_undegraded":"2023-10-11T11:19:01.882635+0000","last_fullsized":"2023-10-11T11:19:01.882635+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T17:21:16.866392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":517,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-10-11T11:19:02.775128+0000","last_change":"2023-10-11T11:15:31.670813+0000","last_active":"2023-10-11T11:19:02.775128+0000","last_peered":"2023-10-11T11:19:02.775128+0000","last_clean":"2023-10-11T11:15:27.638219+0000","last_became_active":"2023-10-11T11:15:31.670813+0000","last_became_peered":"2023-10-11T11:15:31.670813+0000","last_unstale":"2023-10-11T11:19:02.775128+0000","last_undegraded":"2023-10-11T11:19:02.775128+0000","last_fullsized":"2023-10-11T11:15:31.658785+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:41:50.872346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":516,"reported_epoch":499,"state":"active+undersized","last_fresh":"2023-10-11T11:19:00.631668+0000","last_change":"2023-10-11T11:15:31.666378+0000","last_active":"2023-10-11T11:19:00.631668+0000","last_peered":"2023-10-11T11:19:00.631668+0000","last_clean":"2023-10-11T11:15:27.638303+0000","last_became_active":"2023-10-11T11:15:31.666378+0000","last_became_peered":"2023-10-11T11:15:31.666378+0000","last_unstale":"2023-10-11T11:19:00.631668+0000","last_undegraded":"2023-10-11T11:19:00.631668+0000","last_fullsized":"2023-10-11T11:15:31.649186+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T15:35:19.331209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":516,"reported_epoch":499,"state":"active+undersized","last_fresh":"2023-10-11T11:19:00.636960+0000","last_change":"2023-10-11T11:15:31.679465+0000","last_active":"2023-10-11T11:19:00.636960+0000","last_peered":"2023-10-11T11:19:00.636960+0000","last_clean":"2023-10-11T11:15:27.638749+0000","last_became_active":"2023-10-11T11:15:31.679465+0000","last_became_peered":"2023-10-11T11:15:31.679465+0000","last_unstale":"2023-10-11T11:19:00.636960+0000","last_undegraded":"2023-10-11T11:19:00.636960+0000","last_fullsized":"2023-10-11T11:15:31.659135+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T16:03:42.656692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":11,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738801,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":785348,"kb_used_data":5812,"kb_used_omap":96,"kb_used_meta":779423,"kb_avail":93586492,"statfs":{"total":96636764160,"available":95832567808,"internally_reserved":0,"allocated":5951488,"data_stored":3106918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":98609,"internal_metadata":798129871},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 11:14:37 2023","interfaces":[{"interface":"back","average":{"1min":0.314,"5min":0.316,"15min":0.443},"min":{"1min":0.24399999999999999,"5min":0.14799999999999999,"15min":0.14599999999999999},"max":{"1min":0.48499999999999999,"5min":1.712,"15min":9.6400000000000006},"last":0.31},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.35499999999999998,"15min":0.41299999999999998},"min":{"1min":0.20899999999999999,"5min":0.17199999999999999,"15min":0.16400000000000001},"max":{"1min":3.8180000000000001,"5min":3.8180000000000001,"15min":6.3200000000000003},"last":0.26500000000000001}]},{"osd":1,"last update":"Wed Oct 11 11:18:39 2023","interfaces":[{"interface":"back","average":{"1min":0.27500000000000002,"5min":0.29299999999999998,"15min":0.32400000000000001},"min":{"1min":0.154,"5min":0.154,"15min":0.154},"max":{"1min":0.41999999999999998,"5min":0.59299999999999997,"15min":2.9540000000000002},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.307,"5min":0.35899999999999999,"15min":0.36499999999999999},"min":{"1min":0.18099999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.45100000000000001,"5min":3.2719999999999998,"15min":3.2719999999999998},"last":0.38300000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738756,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1484496,"kb_used_data":406792,"kb_used_omap":107,"kb_used_meta":1077588,"kb_avail":92887344,"statfs":{"total":96636764160,"available":95116640256,"internally_reserved":0,"allocated":416555008,"data_stored":413648921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110309,"internal_metadata":1103450395},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Oct 11 11:14:30 2023","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.35399999999999998,"15min":0.40600000000000003},"min":{"1min":0.18099999999999999,"5min":0.155,"15min":0.14899999999999999},"max":{"1min":1.4319999999999999,"5min":2.3460000000000001,"15min":6.593},"last":0.26200000000000001},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.39100000000000001,"15min":0.48199999999999998},"min":{"1min":0.189,"5min":0.189,"15min":0.183},"max":{"1min":2.2120000000000002,"5min":2.4089999999999998,"15min":6.5960000000000001},"last":0.316}]},{"osd":2,"last update":"Wed Oct 11 11:14:30 2023","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.35499999999999998,"15min":0.47099999999999997},"min":{"1min":0.20300000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":1.2490000000000001,"5min":2.665,"15min":8.4730000000000008},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.375,"15min":0.442},"min":{"1min":0.23200000000000001,"5min":0.20499999999999999,"15min":0.19600000000000001},"max":{"1min":1.28,"5min":2.6230000000000002,"15min":7.0129999999999999},"last":0.33600000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738802,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":826440,"kb_used_data":6392,"kb_used_omap":88,"kb_used_meta":819943,"kb_avail":93545400,"statfs":{"total":96636764160,"available":95790489600,"internally_reserved":0,"allocated":6545408,"data_stored":3697286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":90523,"internal_metadata":839622245},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 11:14:33 2023","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.42799999999999999,"15min":0.41099999999999998},"min":{"1min":0.151,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":1.2809999999999999,"5min":3.5739999999999998,"15min":8.2479999999999993},"last":0.223},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.48299999999999998,"15min":0.49199999999999999},"min":{"1min":0.20699999999999999,"5min":0.182,"15min":0.17999999999999999},"max":{"1min":1.323,"5min":3.6080000000000001,"15min":8.2870000000000008},"last":0.28399999999999997}]},{"osd":2,"last update":"Wed Oct 11 11:18:40 2023","interfaces":[{"interface":"back","average":{"1min":0.33400000000000002,"5min":0.378,"15min":0.35599999999999998},"min":{"1min":0.16500000000000001,"5min":0.14699999999999999,"15min":0.14699999999999999},"max":{"1min":1.244,"5min":7.4829999999999997,"15min":10.617000000000001},"last":0.375},{"interface":"front","average":{"1min":0.34599999999999997,"5min":0.39000000000000001,"15min":0.34999999999999998},"min":{"1min":0.20399999999999999,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.81100000000000005,"5min":7.5350000000000001,"15min":7.5350000000000001},"last":0.32300000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-11T11:19:26.667 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-10-11T11:19:26.981 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T11:19:26.983 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2023-10-11T11:19:26.995 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":1574,"stamp":"2023-10-11T11:19:26.649964+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":11,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":107,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3096284,"kb_used_data":418996,"kb_used_omap":292,"kb_used_meta":2676955,"kb_avail":280019236,"statfs":{"total":289910292480,"available":286739697664,"internally_reserved":0,"allocated":429051904,"data_stored":420453125,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299441,"internal_metadata":2741202511},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002917"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":517,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-10-11T11:19:02.772828+0000","last_change":"2023-10-11T11:15:31.661747+0000","last_active":"2023-10-11T11:19:02.772828+0000","last_peered":"2023-10-11T11:19:02.772828+0000","last_clean":"2023-10-11T11:15:27.638215+0000","last_became_active":"2023-10-11T11:15:31.661747+0000","last_became_peered":"2023-10-11T11:15:31.661747+0000","last_unstale":"2023-10-11T11:19:02.772828+0000","last_undegraded":"2023-10-11T11:19:02.772828+0000","last_fullsized":"2023-10-11T11:15:31.649336+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:15:25.047039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":28,"reported_epoch":499,"state":"active+undersized","last_fresh":"2023-10-11T11:19:00.628569+0000","last_change":"2023-10-11T11:15:31.671363+0000","last_active":"2023-10-11T11:19:00.628569+0000","last_peered":"2023-10-11T11:19:00.628569+0000","last_clean":"2023-10-11T10:43:40.751300+0000","last_became_active":"2023-10-11T11:15:31.671363+0000","last_became_peered":"2023-10-11T11:15:31.671363+0000","last_unstale":"2023-10-11T11:19:00.628569+0000","last_undegraded":"2023-10-11T11:19:00.628569+0000","last_fullsized":"2023-10-11T11:15:31.652001+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:01:53.884964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":512,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-10-11T11:19:03.429640+0000","last_change":"2023-10-11T10:43:41.772781+0000","last_active":"2023-10-11T11:19:03.429640+0000","last_peered":"2023-10-11T11:19:03.429640+0000","last_clean":"2023-10-11T11:19:03.429640+0000","last_became_active":"2023-10-11T10:43:41.772443+0000","last_became_peered":"2023-10-11T10:43:41.772443+0000","last_unstale":"2023-10-11T11:19:03.429640+0000","last_undegraded":"2023-10-11T11:19:03.429640+0000","last_fullsized":"2023-10-11T11:19:03.429640+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T14:15:30.547284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":513,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-10-11T11:19:02.772211+0000","last_change":"2023-10-11T10:43:41.772615+0000","last_active":"2023-10-11T11:19:02.772211+0000","last_peered":"2023-10-11T11:19:02.772211+0000","last_clean":"2023-10-11T11:19:02.772211+0000","last_became_active":"2023-10-11T10:43:41.772290+0000","last_became_peered":"2023-10-11T10:43:41.772290+0000","last_unstale":"2023-10-11T11:19:02.772211+0000","last_undegraded":"2023-10-11T11:19:02.772211+0000","last_fullsized":"2023-10-11T11:19:02.772211+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:26:06.909963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":647,"reported_epoch":499,"state":"active+undersized+degraded","last_fresh":"2023-10-11T11:19:00.625113+0000","last_change":"2023-10-11T11:15:31.654022+0000","last_active":"2023-10-11T11:19:00.625113+0000","last_peered":"2023-10-11T11:19:00.625113+0000","last_clean":"2023-10-11T11:15:27.636884+0000","last_became_active":"2023-10-11T11:15:31.654022+0000","last_became_peered":"2023-10-11T11:15:31.654022+0000","last_unstale":"2023-10-11T11:19:00.625113+0000","last_undegraded":"2023-10-11T11:15:31.646946+0000","last_fullsized":"2023-10-11T11:15:31.646751+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:36.739637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:36.739637+0000","last_clean_scrub_stamp":"2023-10-11T10:43:36.739637+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T11:33:33.696447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":519,"reported_epoch":499,"state":"active+clean+remapped","last_fresh":"2023-10-11T11:19:01.882635+0000","last_change":"2023-10-11T11:15:33.660200+0000","last_active":"2023-10-11T11:19:01.882635+0000","last_peered":"2023-10-11T11:19:01.882635+0000","last_clean":"2023-10-11T11:19:01.882635+0000","last_became_active":"2023-10-11T11:15:33.659619+0000","last_became_peered":"2023-10-11T11:15:33.659619+0000","last_unstale":"2023-10-11T11:19:01.882635+0000","last_undegraded":"2023-10-11T11:19:01.882635+0000","last_fullsized":"2023-10-11T11:19:01.882635+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T17:21:16.866392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":517,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-10-11T11:19:02.775128+0000","last_change":"2023-10-11T11:15:31.670813+0000","last_active":"2023-10-11T11:19:02.775128+0000","last_peered":"2023-10-11T11:19:02.775128+0000","last_clean":"2023-10-11T11:15:27.638219+0000","last_became_active":"2023-10-11T11:15:31.670813+0000","last_became_peered":"2023-10-11T11:15:31.670813+0000","last_unstale":"2023-10-11T11:19:02.775128+0000","last_undegraded":"2023-10-11T11:19:02.775128+0000","last_fullsized":"2023-10-11T11:15:31.658785+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:41:50.872346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":516,"reported_epoch":499,"state":"active+undersized","last_fresh":"2023-10-11T11:19:00.631668+0000","last_change":"2023-10-11T11:15:31.666378+0000","last_active":"2023-10-11T11:19:00.631668+0000","last_peered":"2023-10-11T11:19:00.631668+0000","last_clean":"2023-10-11T11:15:27.638303+0000","last_became_active":"2023-10-11T11:15:31.666378+0000","last_became_peered":"2023-10-11T11:15:31.666378+0000","last_unstale":"2023-10-11T11:19:00.631668+0000","last_undegraded":"2023-10-11T11:19:00.631668+0000","last_fullsized":"2023-10-11T11:15:31.649186+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T15:35:19.331209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":516,"reported_epoch":499,"state":"active+undersized","last_fresh":"2023-10-11T11:19:00.636960+0000","last_change":"2023-10-11T11:15:31.679465+0000","last_active":"2023-10-11T11:19:00.636960+0000","last_peered":"2023-10-11T11:19:00.636960+0000","last_clean":"2023-10-11T11:15:27.638749+0000","last_became_active":"2023-10-11T11:15:31.679465+0000","last_became_peered":"2023-10-11T11:15:31.679465+0000","last_unstale":"2023-10-11T11:19:00.636960+0000","last_undegraded":"2023-10-11T11:19:00.636960+0000","last_fullsized":"2023-10-11T11:15:31.659135+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T16:03:42.656692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":11,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738801,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":785348,"kb_used_data":5812,"kb_used_omap":96,"kb_used_meta":779423,"kb_avail":93586492,"statfs":{"total":96636764160,"available":95832567808,"internally_reserved":0,"allocated":5951488,"data_stored":3106918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":98609,"internal_metadata":798129871},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 11:14:37 2023","interfaces":[{"interface":"back","average":{"1min":0.314,"5min":0.316,"15min":0.443},"min":{"1min":0.24399999999999999,"5min":0.14799999999999999,"15min":0.14599999999999999},"max":{"1min":0.48499999999999999,"5min":1.712,"15min":9.6400000000000006},"last":0.31},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.35499999999999998,"15min":0.41299999999999998},"min":{"1min":0.20899999999999999,"5min":0.17199999999999999,"15min":0.16400000000000001},"max":{"1min":3.8180000000000001,"5min":3.8180000000000001,"15min":6.3200000000000003},"last":0.26500000000000001}]},{"osd":1,"last update":"Wed Oct 11 11:18:39 2023","interfaces":[{"interface":"back","average":{"1min":0.27500000000000002,"5min":0.29299999999999998,"15min":0.32400000000000001},"min":{"1min":0.154,"5min":0.154,"15min":0.154},"max":{"1min":0.41999999999999998,"5min":0.59299999999999997,"15min":2.9540000000000002},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.307,"5min":0.35899999999999999,"15min":0.36499999999999999},"min":{"1min":0.18099999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.45100000000000001,"5min":3.2719999999999998,"15min":3.2719999999999998},"last":0.38300000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738756,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1484496,"kb_used_data":406792,"kb_used_omap":107,"kb_used_meta":1077588,"kb_avail":92887344,"statfs":{"total":96636764160,"available":95116640256,"internally_reserved":0,"allocated":416555008,"data_stored":413648921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110309,"internal_metadata":1103450395},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Oct 11 11:14:30 2023","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.35399999999999998,"15min":0.40600000000000003},"min":{"1min":0.18099999999999999,"5min":0.155,"15min":0.14899999999999999},"max":{"1min":1.4319999999999999,"5min":2.3460000000000001,"15min":6.593},"last":0.26200000000000001},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.39100000000000001,"15min":0.48199999999999998},"min":{"1min":0.189,"5min":0.189,"15min":0.183},"max":{"1min":2.2120000000000002,"5min":2.4089999999999998,"15min":6.5960000000000001},"last":0.316}]},{"osd":2,"last update":"Wed Oct 11 11:14:30 2023","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.35499999999999998,"15min":0.47099999999999997},"min":{"1min":0.20300000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":1.2490000000000001,"5min":2.665,"15min":8.4730000000000008},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.375,"15min":0.442},"min":{"1min":0.23200000000000001,"5min":0.20499999999999999,"15min":0.19600000000000001},"max":{"1min":1.28,"5min":2.6230000000000002,"15min":7.0129999999999999},"last":0.33600000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738802,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":826440,"kb_used_data":6392,"kb_used_omap":88,"kb_used_meta":819943,"kb_avail":93545400,"statfs":{"total":96636764160,"available":95790489600,"internally_reserved":0,"allocated":6545408,"data_stored":3697286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":90523,"internal_metadata":839622245},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 11:14:33 2023","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.42799999999999999,"15min":0.41099999999999998},"min":{"1min":0.151,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":1.2809999999999999,"5min":3.5739999999999998,"15min":8.2479999999999993},"last":0.223},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.48299999999999998,"15min":0.49199999999999999},"min":{"1min":0.20699999999999999,"5min":0.182,"15min":0.17999999999999999},"max":{"1min":1.323,"5min":3.6080000000000001,"15min":8.2870000000000008},"last":0.28399999999999997}]},{"osd":2,"last update":"Wed Oct 11 11:18:40 2023","interfaces":[{"interface":"back","average":{"1min":0.33400000000000002,"5min":0.378,"15min":0.35599999999999998},"min":{"1min":0.16500000000000001,"5min":0.14699999999999999,"15min":0.14699999999999999},"max":{"1min":1.244,"5min":7.4829999999999997,"15min":10.617000000000001},"last":0.375},{"interface":"front","average":{"1min":0.34599999999999997,"5min":0.39000000000000001,"15min":0.34999999999999998},"min":{"1min":0.20399999999999999,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.81100000000000005,"5min":7.5350000000000001,"15min":7.5350000000000001},"last":0.32300000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-11T11:19:26.996 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-10-11T11:19:26.996 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 517, 'reported_epoch': 500, 'state': 'active+undersized', 'last_fresh': '2023-10-11T11:19:02.772828+0000', 'last_change': '2023-10-11T11:15:31.661747+0000', 'last_active': '2023-10-11T11:19:02.772828+0000', 'last_peered': '2023-10-11T11:19:02.772828+0000', 'last_clean': '2023-10-11T11:15:27.638215+0000', 'last_became_active': '2023-10-11T11:15:31.661747+0000', 'last_became_peered': '2023-10-11T11:15:31.661747+0000', 'last_unstale': '2023-10-11T11:19:02.772828+0000', 'last_undegraded': '2023-10-11T11:19:02.772828+0000', 'last_fullsized': '2023-10-11T11:15:31.649336+0000', 'mapping_epoch': 480, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_clean_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-12T22:15:25.047039+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-10-11T11:19:26.996 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-10-11T11:19:26.997 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 28, 'reported_epoch': 499, 'state': 'active+undersized', 'last_fresh': '2023-10-11T11:19:00.628569+0000', 'last_change': '2023-10-11T11:15:31.671363+0000', 'last_active': '2023-10-11T11:19:00.628569+0000', 'last_peered': '2023-10-11T11:19:00.628569+0000', 'last_clean': '2023-10-11T10:43:40.751300+0000', 'last_became_active': '2023-10-11T11:15:31.671363+0000', 'last_became_peered': '2023-10-11T11:15:31.671363+0000', 'last_unstale': '2023-10-11T11:19:00.628569+0000', 'last_undegraded': '2023-10-11T11:19:00.628569+0000', 'last_fullsized': '2023-10-11T11:15:31.652001+0000', 'mapping_epoch': 480, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_clean_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-12T21:01:53.884964+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-10-11T11:19:26.997 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-10-11T11:19:26.997 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "11'97", 'reported_seq': 647, 'reported_epoch': 499, 'state': 'active+undersized+degraded', 'last_fresh': '2023-10-11T11:19:00.625113+0000', 'last_change': '2023-10-11T11:15:31.654022+0000', 'last_active': '2023-10-11T11:19:00.625113+0000', 'last_peered': '2023-10-11T11:19:00.625113+0000', 'last_clean': '2023-10-11T11:15:27.636884+0000', 'last_became_active': '2023-10-11T11:15:31.654022+0000', 'last_became_peered': '2023-10-11T11:15:31.654022+0000', 'last_unstale': '2023-10-11T11:19:00.625113+0000', 'last_undegraded': '2023-10-11T11:15:31.646946+0000', 'last_fullsized': '2023-10-11T11:15:31.646751+0000', 'mapping_epoch': 480, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-11T10:43:36.739637+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-11T10:43:36.739637+0000', 'last_clean_scrub_stamp': '2023-10-11T10:43:36.739637+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-12T11:33:33.696447+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-10-11T11:19:26.998 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.0 is not active+clean 2023-10-11T11:19:26.998 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 519, 'reported_epoch': 499, 'state': 'active+clean+remapped', 'last_fresh': '2023-10-11T11:19:01.882635+0000', 'last_change': '2023-10-11T11:15:33.660200+0000', 'last_active': '2023-10-11T11:19:01.882635+0000', 'last_peered': '2023-10-11T11:19:01.882635+0000', 'last_clean': '2023-10-11T11:19:01.882635+0000', 'last_became_active': '2023-10-11T11:15:33.659619+0000', 'last_became_peered': '2023-10-11T11:15:33.659619+0000', 'last_unstale': '2023-10-11T11:19:01.882635+0000', 'last_undegraded': '2023-10-11T11:19:01.882635+0000', 'last_fullsized': '2023-10-11T11:19:01.882635+0000', 'mapping_epoch': 482, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 483, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_clean_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-12T17:21:16.866392+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2, 1], 'avail_no_missing': ['2', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-10-11T11:19:26.998 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-10-11T11:19:26.999 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 517, 'reported_epoch': 500, 'state': 'active+undersized', 'last_fresh': '2023-10-11T11:19:02.775128+0000', 'last_change': '2023-10-11T11:15:31.670813+0000', 'last_active': '2023-10-11T11:19:02.775128+0000', 'last_peered': '2023-10-11T11:19:02.775128+0000', 'last_clean': '2023-10-11T11:15:27.638219+0000', 'last_became_active': '2023-10-11T11:15:31.670813+0000', 'last_became_peered': '2023-10-11T11:15:31.670813+0000', 'last_unstale': '2023-10-11T11:19:02.775128+0000', 'last_undegraded': '2023-10-11T11:19:02.775128+0000', 'last_fullsized': '2023-10-11T11:15:31.658785+0000', 'mapping_epoch': 480, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_clean_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-12T21:41:50.872346+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-10-11T11:19:26.999 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-10-11T11:19:26.999 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 516, 'reported_epoch': 499, 'state': 'active+undersized', 'last_fresh': '2023-10-11T11:19:00.631668+0000', 'last_change': '2023-10-11T11:15:31.666378+0000', 'last_active': '2023-10-11T11:19:00.631668+0000', 'last_peered': '2023-10-11T11:19:00.631668+0000', 'last_clean': '2023-10-11T11:15:27.638303+0000', 'last_became_active': '2023-10-11T11:15:31.666378+0000', 'last_became_peered': '2023-10-11T11:15:31.666378+0000', 'last_unstale': '2023-10-11T11:19:00.631668+0000', 'last_undegraded': '2023-10-11T11:19:00.631668+0000', 'last_fullsized': '2023-10-11T11:15:31.649186+0000', 'mapping_epoch': 480, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_clean_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-12T15:35:19.331209+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-10-11T11:19:27.000 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-10-11T11:19:27.000 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 516, 'reported_epoch': 499, 'state': 'active+undersized', 'last_fresh': '2023-10-11T11:19:00.636960+0000', 'last_change': '2023-10-11T11:15:31.679465+0000', 'last_active': '2023-10-11T11:19:00.636960+0000', 'last_peered': '2023-10-11T11:19:00.636960+0000', 'last_clean': '2023-10-11T11:15:27.638749+0000', 'last_became_active': '2023-10-11T11:15:31.679465+0000', 'last_became_peered': '2023-10-11T11:15:31.679465+0000', 'last_unstale': '2023-10-11T11:19:00.636960+0000', 'last_undegraded': '2023-10-11T11:19:00.636960+0000', 'last_fullsized': '2023-10-11T11:15:31.659135+0000', 'mapping_epoch': 480, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_clean_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-12T16:03:42.656692+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-10-11T11:19:27.001 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-10-11T11:19:27.395 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T11:19:27.395 INFO:teuthology.orchestra.run.smithi111.stdout:{"fsid":"022ce6f2-9a5b-45ad-8b77-a1332d92f228","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2162,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":500,"num_osds":3,"num_up_osds":2,"osd_up_since":1697022930,"num_in_osds":3,"osd_in_since":1697021005,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":2},{"state_name":"active+undersized+degraded","count":1},{"state_name":"active+clean+remapped","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3170594816,"bytes_avail":286739697664,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":36,"modified":"2023-10-11T11:19:04.644408+0000","services":{}},"progress_events":{}} 2023-10-11T11:19:27.409 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-10-11T11:19:27.409 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-10-11T11:19:27.726 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T11:19:27.729 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2023-10-11T11:19:27.740 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":1574,"stamp":"2023-10-11T11:19:26.649964+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":11,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":107,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3096284,"kb_used_data":418996,"kb_used_omap":292,"kb_used_meta":2676955,"kb_avail":280019236,"statfs":{"total":289910292480,"available":286739697664,"internally_reserved":0,"allocated":429051904,"data_stored":420453125,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299441,"internal_metadata":2741202511},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002917"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":517,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-10-11T11:19:02.772828+0000","last_change":"2023-10-11T11:15:31.661747+0000","last_active":"2023-10-11T11:19:02.772828+0000","last_peered":"2023-10-11T11:19:02.772828+0000","last_clean":"2023-10-11T11:15:27.638215+0000","last_became_active":"2023-10-11T11:15:31.661747+0000","last_became_peered":"2023-10-11T11:15:31.661747+0000","last_unstale":"2023-10-11T11:19:02.772828+0000","last_undegraded":"2023-10-11T11:19:02.772828+0000","last_fullsized":"2023-10-11T11:15:31.649336+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:15:25.047039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":28,"reported_epoch":499,"state":"active+undersized","last_fresh":"2023-10-11T11:19:00.628569+0000","last_change":"2023-10-11T11:15:31.671363+0000","last_active":"2023-10-11T11:19:00.628569+0000","last_peered":"2023-10-11T11:19:00.628569+0000","last_clean":"2023-10-11T10:43:40.751300+0000","last_became_active":"2023-10-11T11:15:31.671363+0000","last_became_peered":"2023-10-11T11:15:31.671363+0000","last_unstale":"2023-10-11T11:19:00.628569+0000","last_undegraded":"2023-10-11T11:19:00.628569+0000","last_fullsized":"2023-10-11T11:15:31.652001+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:01:53.884964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":512,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-10-11T11:19:03.429640+0000","last_change":"2023-10-11T10:43:41.772781+0000","last_active":"2023-10-11T11:19:03.429640+0000","last_peered":"2023-10-11T11:19:03.429640+0000","last_clean":"2023-10-11T11:19:03.429640+0000","last_became_active":"2023-10-11T10:43:41.772443+0000","last_became_peered":"2023-10-11T10:43:41.772443+0000","last_unstale":"2023-10-11T11:19:03.429640+0000","last_undegraded":"2023-10-11T11:19:03.429640+0000","last_fullsized":"2023-10-11T11:19:03.429640+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T14:15:30.547284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":513,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-10-11T11:19:02.772211+0000","last_change":"2023-10-11T10:43:41.772615+0000","last_active":"2023-10-11T11:19:02.772211+0000","last_peered":"2023-10-11T11:19:02.772211+0000","last_clean":"2023-10-11T11:19:02.772211+0000","last_became_active":"2023-10-11T10:43:41.772290+0000","last_became_peered":"2023-10-11T10:43:41.772290+0000","last_unstale":"2023-10-11T11:19:02.772211+0000","last_undegraded":"2023-10-11T11:19:02.772211+0000","last_fullsized":"2023-10-11T11:19:02.772211+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:26:06.909963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":647,"reported_epoch":499,"state":"active+undersized+degraded","last_fresh":"2023-10-11T11:19:00.625113+0000","last_change":"2023-10-11T11:15:31.654022+0000","last_active":"2023-10-11T11:19:00.625113+0000","last_peered":"2023-10-11T11:19:00.625113+0000","last_clean":"2023-10-11T11:15:27.636884+0000","last_became_active":"2023-10-11T11:15:31.654022+0000","last_became_peered":"2023-10-11T11:15:31.654022+0000","last_unstale":"2023-10-11T11:19:00.625113+0000","last_undegraded":"2023-10-11T11:15:31.646946+0000","last_fullsized":"2023-10-11T11:15:31.646751+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:36.739637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:36.739637+0000","last_clean_scrub_stamp":"2023-10-11T10:43:36.739637+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T11:33:33.696447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":519,"reported_epoch":499,"state":"active+clean+remapped","last_fresh":"2023-10-11T11:19:01.882635+0000","last_change":"2023-10-11T11:15:33.660200+0000","last_active":"2023-10-11T11:19:01.882635+0000","last_peered":"2023-10-11T11:19:01.882635+0000","last_clean":"2023-10-11T11:19:01.882635+0000","last_became_active":"2023-10-11T11:15:33.659619+0000","last_became_peered":"2023-10-11T11:15:33.659619+0000","last_unstale":"2023-10-11T11:19:01.882635+0000","last_undegraded":"2023-10-11T11:19:01.882635+0000","last_fullsized":"2023-10-11T11:19:01.882635+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T17:21:16.866392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":517,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-10-11T11:19:02.775128+0000","last_change":"2023-10-11T11:15:31.670813+0000","last_active":"2023-10-11T11:19:02.775128+0000","last_peered":"2023-10-11T11:19:02.775128+0000","last_clean":"2023-10-11T11:15:27.638219+0000","last_became_active":"2023-10-11T11:15:31.670813+0000","last_became_peered":"2023-10-11T11:15:31.670813+0000","last_unstale":"2023-10-11T11:19:02.775128+0000","last_undegraded":"2023-10-11T11:19:02.775128+0000","last_fullsized":"2023-10-11T11:15:31.658785+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:41:50.872346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":516,"reported_epoch":499,"state":"active+undersized","last_fresh":"2023-10-11T11:19:00.631668+0000","last_change":"2023-10-11T11:15:31.666378+0000","last_active":"2023-10-11T11:19:00.631668+0000","last_peered":"2023-10-11T11:19:00.631668+0000","last_clean":"2023-10-11T11:15:27.638303+0000","last_became_active":"2023-10-11T11:15:31.666378+0000","last_became_peered":"2023-10-11T11:15:31.666378+0000","last_unstale":"2023-10-11T11:19:00.631668+0000","last_undegraded":"2023-10-11T11:19:00.631668+0000","last_fullsized":"2023-10-11T11:15:31.649186+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T15:35:19.331209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":516,"reported_epoch":499,"state":"active+undersized","last_fresh":"2023-10-11T11:19:00.636960+0000","last_change":"2023-10-11T11:15:31.679465+0000","last_active":"2023-10-11T11:19:00.636960+0000","last_peered":"2023-10-11T11:19:00.636960+0000","last_clean":"2023-10-11T11:15:27.638749+0000","last_became_active":"2023-10-11T11:15:31.679465+0000","last_became_peered":"2023-10-11T11:15:31.679465+0000","last_unstale":"2023-10-11T11:19:00.636960+0000","last_undegraded":"2023-10-11T11:19:00.636960+0000","last_fullsized":"2023-10-11T11:15:31.659135+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T16:03:42.656692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":11,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738801,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":785348,"kb_used_data":5812,"kb_used_omap":96,"kb_used_meta":779423,"kb_avail":93586492,"statfs":{"total":96636764160,"available":95832567808,"internally_reserved":0,"allocated":5951488,"data_stored":3106918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":98609,"internal_metadata":798129871},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 11:14:37 2023","interfaces":[{"interface":"back","average":{"1min":0.314,"5min":0.316,"15min":0.443},"min":{"1min":0.24399999999999999,"5min":0.14799999999999999,"15min":0.14599999999999999},"max":{"1min":0.48499999999999999,"5min":1.712,"15min":9.6400000000000006},"last":0.31},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.35499999999999998,"15min":0.41299999999999998},"min":{"1min":0.20899999999999999,"5min":0.17199999999999999,"15min":0.16400000000000001},"max":{"1min":3.8180000000000001,"5min":3.8180000000000001,"15min":6.3200000000000003},"last":0.26500000000000001}]},{"osd":1,"last update":"Wed Oct 11 11:18:39 2023","interfaces":[{"interface":"back","average":{"1min":0.27500000000000002,"5min":0.29299999999999998,"15min":0.32400000000000001},"min":{"1min":0.154,"5min":0.154,"15min":0.154},"max":{"1min":0.41999999999999998,"5min":0.59299999999999997,"15min":2.9540000000000002},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.307,"5min":0.35899999999999999,"15min":0.36499999999999999},"min":{"1min":0.18099999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.45100000000000001,"5min":3.2719999999999998,"15min":3.2719999999999998},"last":0.38300000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738756,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1484496,"kb_used_data":406792,"kb_used_omap":107,"kb_used_meta":1077588,"kb_avail":92887344,"statfs":{"total":96636764160,"available":95116640256,"internally_reserved":0,"allocated":416555008,"data_stored":413648921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110309,"internal_metadata":1103450395},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Oct 11 11:14:30 2023","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.35399999999999998,"15min":0.40600000000000003},"min":{"1min":0.18099999999999999,"5min":0.155,"15min":0.14899999999999999},"max":{"1min":1.4319999999999999,"5min":2.3460000000000001,"15min":6.593},"last":0.26200000000000001},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.39100000000000001,"15min":0.48199999999999998},"min":{"1min":0.189,"5min":0.189,"15min":0.183},"max":{"1min":2.2120000000000002,"5min":2.4089999999999998,"15min":6.5960000000000001},"last":0.316}]},{"osd":2,"last update":"Wed Oct 11 11:14:30 2023","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.35499999999999998,"15min":0.47099999999999997},"min":{"1min":0.20300000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":1.2490000000000001,"5min":2.665,"15min":8.4730000000000008},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.375,"15min":0.442},"min":{"1min":0.23200000000000001,"5min":0.20499999999999999,"15min":0.19600000000000001},"max":{"1min":1.28,"5min":2.6230000000000002,"15min":7.0129999999999999},"last":0.33600000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738802,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":826440,"kb_used_data":6392,"kb_used_omap":88,"kb_used_meta":819943,"kb_avail":93545400,"statfs":{"total":96636764160,"available":95790489600,"internally_reserved":0,"allocated":6545408,"data_stored":3697286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":90523,"internal_metadata":839622245},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 11:14:33 2023","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.42799999999999999,"15min":0.41099999999999998},"min":{"1min":0.151,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":1.2809999999999999,"5min":3.5739999999999998,"15min":8.2479999999999993},"last":0.223},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.48299999999999998,"15min":0.49199999999999999},"min":{"1min":0.20699999999999999,"5min":0.182,"15min":0.17999999999999999},"max":{"1min":1.323,"5min":3.6080000000000001,"15min":8.2870000000000008},"last":0.28399999999999997}]},{"osd":2,"last update":"Wed Oct 11 11:18:40 2023","interfaces":[{"interface":"back","average":{"1min":0.33400000000000002,"5min":0.378,"15min":0.35599999999999998},"min":{"1min":0.16500000000000001,"5min":0.14699999999999999,"15min":0.14699999999999999},"max":{"1min":1.244,"5min":7.4829999999999997,"15min":10.617000000000001},"last":0.375},{"interface":"front","average":{"1min":0.34599999999999997,"5min":0.39000000000000001,"15min":0.34999999999999998},"min":{"1min":0.20399999999999999,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.81100000000000005,"5min":7.5350000000000001,"15min":7.5350000000000001},"last":0.32300000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-11T11:19:29.927 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~238s 2023-10-11T11:19:30.742 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-10-11T11:19:31.073 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T11:19:31.075 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2023-10-11T11:19:31.086 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":1576,"stamp":"2023-10-11T11:19:30.650916+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":11,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":107,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3096284,"kb_used_data":418996,"kb_used_omap":292,"kb_used_meta":2676955,"kb_avail":280019236,"statfs":{"total":289910292480,"available":286739697664,"internally_reserved":0,"allocated":429051904,"data_stored":420453125,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299441,"internal_metadata":2741202511},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002736"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":517,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-10-11T11:19:02.772828+0000","last_change":"2023-10-11T11:15:31.661747+0000","last_active":"2023-10-11T11:19:02.772828+0000","last_peered":"2023-10-11T11:19:02.772828+0000","last_clean":"2023-10-11T11:15:27.638215+0000","last_became_active":"2023-10-11T11:15:31.661747+0000","last_became_peered":"2023-10-11T11:15:31.661747+0000","last_unstale":"2023-10-11T11:19:02.772828+0000","last_undegraded":"2023-10-11T11:19:02.772828+0000","last_fullsized":"2023-10-11T11:15:31.649336+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:15:25.047039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":28,"reported_epoch":499,"state":"active+undersized","last_fresh":"2023-10-11T11:19:00.628569+0000","last_change":"2023-10-11T11:15:31.671363+0000","last_active":"2023-10-11T11:19:00.628569+0000","last_peered":"2023-10-11T11:19:00.628569+0000","last_clean":"2023-10-11T10:43:40.751300+0000","last_became_active":"2023-10-11T11:15:31.671363+0000","last_became_peered":"2023-10-11T11:15:31.671363+0000","last_unstale":"2023-10-11T11:19:00.628569+0000","last_undegraded":"2023-10-11T11:19:00.628569+0000","last_fullsized":"2023-10-11T11:15:31.652001+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:01:53.884964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":512,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-10-11T11:19:03.429640+0000","last_change":"2023-10-11T10:43:41.772781+0000","last_active":"2023-10-11T11:19:03.429640+0000","last_peered":"2023-10-11T11:19:03.429640+0000","last_clean":"2023-10-11T11:19:03.429640+0000","last_became_active":"2023-10-11T10:43:41.772443+0000","last_became_peered":"2023-10-11T10:43:41.772443+0000","last_unstale":"2023-10-11T11:19:03.429640+0000","last_undegraded":"2023-10-11T11:19:03.429640+0000","last_fullsized":"2023-10-11T11:19:03.429640+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T14:15:30.547284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":513,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-10-11T11:19:02.772211+0000","last_change":"2023-10-11T10:43:41.772615+0000","last_active":"2023-10-11T11:19:02.772211+0000","last_peered":"2023-10-11T11:19:02.772211+0000","last_clean":"2023-10-11T11:19:02.772211+0000","last_became_active":"2023-10-11T10:43:41.772290+0000","last_became_peered":"2023-10-11T10:43:41.772290+0000","last_unstale":"2023-10-11T11:19:02.772211+0000","last_undegraded":"2023-10-11T11:19:02.772211+0000","last_fullsized":"2023-10-11T11:19:02.772211+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:26:06.909963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":647,"reported_epoch":499,"state":"active+undersized+degraded","last_fresh":"2023-10-11T11:19:00.625113+0000","last_change":"2023-10-11T11:15:31.654022+0000","last_active":"2023-10-11T11:19:00.625113+0000","last_peered":"2023-10-11T11:19:00.625113+0000","last_clean":"2023-10-11T11:15:27.636884+0000","last_became_active":"2023-10-11T11:15:31.654022+0000","last_became_peered":"2023-10-11T11:15:31.654022+0000","last_unstale":"2023-10-11T11:19:00.625113+0000","last_undegraded":"2023-10-11T11:15:31.646946+0000","last_fullsized":"2023-10-11T11:15:31.646751+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:36.739637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:36.739637+0000","last_clean_scrub_stamp":"2023-10-11T10:43:36.739637+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T11:33:33.696447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":519,"reported_epoch":499,"state":"active+clean+remapped","last_fresh":"2023-10-11T11:19:01.882635+0000","last_change":"2023-10-11T11:15:33.660200+0000","last_active":"2023-10-11T11:19:01.882635+0000","last_peered":"2023-10-11T11:19:01.882635+0000","last_clean":"2023-10-11T11:19:01.882635+0000","last_became_active":"2023-10-11T11:15:33.659619+0000","last_became_peered":"2023-10-11T11:15:33.659619+0000","last_unstale":"2023-10-11T11:19:01.882635+0000","last_undegraded":"2023-10-11T11:19:01.882635+0000","last_fullsized":"2023-10-11T11:19:01.882635+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T17:21:16.866392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":517,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-10-11T11:19:02.775128+0000","last_change":"2023-10-11T11:15:31.670813+0000","last_active":"2023-10-11T11:19:02.775128+0000","last_peered":"2023-10-11T11:19:02.775128+0000","last_clean":"2023-10-11T11:15:27.638219+0000","last_became_active":"2023-10-11T11:15:31.670813+0000","last_became_peered":"2023-10-11T11:15:31.670813+0000","last_unstale":"2023-10-11T11:19:02.775128+0000","last_undegraded":"2023-10-11T11:19:02.775128+0000","last_fullsized":"2023-10-11T11:15:31.658785+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:41:50.872346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":516,"reported_epoch":499,"state":"active+undersized","last_fresh":"2023-10-11T11:19:00.631668+0000","last_change":"2023-10-11T11:15:31.666378+0000","last_active":"2023-10-11T11:19:00.631668+0000","last_peered":"2023-10-11T11:19:00.631668+0000","last_clean":"2023-10-11T11:15:27.638303+0000","last_became_active":"2023-10-11T11:15:31.666378+0000","last_became_peered":"2023-10-11T11:15:31.666378+0000","last_unstale":"2023-10-11T11:19:00.631668+0000","last_undegraded":"2023-10-11T11:19:00.631668+0000","last_fullsized":"2023-10-11T11:15:31.649186+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T15:35:19.331209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":516,"reported_epoch":499,"state":"active+undersized","last_fresh":"2023-10-11T11:19:00.636960+0000","last_change":"2023-10-11T11:15:31.679465+0000","last_active":"2023-10-11T11:19:00.636960+0000","last_peered":"2023-10-11T11:19:00.636960+0000","last_clean":"2023-10-11T11:15:27.638749+0000","last_became_active":"2023-10-11T11:15:31.679465+0000","last_became_peered":"2023-10-11T11:15:31.679465+0000","last_unstale":"2023-10-11T11:19:00.636960+0000","last_undegraded":"2023-10-11T11:19:00.636960+0000","last_fullsized":"2023-10-11T11:15:31.659135+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T16:03:42.656692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":11,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738802,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":785348,"kb_used_data":5812,"kb_used_omap":96,"kb_used_meta":779423,"kb_avail":93586492,"statfs":{"total":96636764160,"available":95832567808,"internally_reserved":0,"allocated":5951488,"data_stored":3106918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":98609,"internal_metadata":798129871},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 11:14:37 2023","interfaces":[{"interface":"back","average":{"1min":0.314,"5min":0.316,"15min":0.443},"min":{"1min":0.24399999999999999,"5min":0.14799999999999999,"15min":0.14599999999999999},"max":{"1min":0.48499999999999999,"5min":1.712,"15min":9.6400000000000006},"last":0.31},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.35499999999999998,"15min":0.41299999999999998},"min":{"1min":0.20899999999999999,"5min":0.17199999999999999,"15min":0.16400000000000001},"max":{"1min":3.8180000000000001,"5min":3.8180000000000001,"15min":6.3200000000000003},"last":0.26500000000000001}]},{"osd":1,"last update":"Wed Oct 11 11:18:39 2023","interfaces":[{"interface":"back","average":{"1min":0.27500000000000002,"5min":0.29299999999999998,"15min":0.32400000000000001},"min":{"1min":0.154,"5min":0.154,"15min":0.154},"max":{"1min":0.41999999999999998,"5min":0.59299999999999997,"15min":2.9540000000000002},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.307,"5min":0.35899999999999999,"15min":0.36499999999999999},"min":{"1min":0.18099999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.45100000000000001,"5min":3.2719999999999998,"15min":3.2719999999999998},"last":0.45000000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738756,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1484496,"kb_used_data":406792,"kb_used_omap":107,"kb_used_meta":1077588,"kb_avail":92887344,"statfs":{"total":96636764160,"available":95116640256,"internally_reserved":0,"allocated":416555008,"data_stored":413648921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110309,"internal_metadata":1103450395},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Oct 11 11:14:30 2023","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.35399999999999998,"15min":0.40600000000000003},"min":{"1min":0.18099999999999999,"5min":0.155,"15min":0.14899999999999999},"max":{"1min":1.4319999999999999,"5min":2.3460000000000001,"15min":6.593},"last":0.26200000000000001},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.39100000000000001,"15min":0.48199999999999998},"min":{"1min":0.189,"5min":0.189,"15min":0.183},"max":{"1min":2.2120000000000002,"5min":2.4089999999999998,"15min":6.5960000000000001},"last":0.316}]},{"osd":2,"last update":"Wed Oct 11 11:14:30 2023","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.35499999999999998,"15min":0.47099999999999997},"min":{"1min":0.20300000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":1.2490000000000001,"5min":2.665,"15min":8.4730000000000008},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.375,"15min":0.442},"min":{"1min":0.23200000000000001,"5min":0.20499999999999999,"15min":0.19600000000000001},"max":{"1min":1.28,"5min":2.6230000000000002,"15min":7.0129999999999999},"last":0.33600000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738803,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":826440,"kb_used_data":6392,"kb_used_omap":88,"kb_used_meta":819943,"kb_avail":93545400,"statfs":{"total":96636764160,"available":95790489600,"internally_reserved":0,"allocated":6545408,"data_stored":3697286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":90523,"internal_metadata":839622245},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 11:14:33 2023","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.42799999999999999,"15min":0.41099999999999998},"min":{"1min":0.151,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":1.2809999999999999,"5min":3.5739999999999998,"15min":8.2479999999999993},"last":0.223},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.48299999999999998,"15min":0.49199999999999999},"min":{"1min":0.20699999999999999,"5min":0.182,"15min":0.17999999999999999},"max":{"1min":1.323,"5min":3.6080000000000001,"15min":8.2870000000000008},"last":0.28399999999999997}]},{"osd":2,"last update":"Wed Oct 11 11:18:40 2023","interfaces":[{"interface":"back","average":{"1min":0.33400000000000002,"5min":0.378,"15min":0.35599999999999998},"min":{"1min":0.16500000000000001,"5min":0.14699999999999999,"15min":0.14699999999999999},"max":{"1min":1.244,"5min":7.4829999999999997,"15min":10.617000000000001},"last":0.371},{"interface":"front","average":{"1min":0.34599999999999997,"5min":0.39000000000000001,"15min":0.34999999999999998},"min":{"1min":0.20399999999999999,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.81100000000000005,"5min":7.5350000000000001,"15min":7.5350000000000001},"last":0.41999999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-11T11:19:31.087 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-10-11T11:19:31.403 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T11:19:31.405 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2023-10-11T11:19:31.417 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":1576,"stamp":"2023-10-11T11:19:30.650916+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":11,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":107,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3096284,"kb_used_data":418996,"kb_used_omap":292,"kb_used_meta":2676955,"kb_avail":280019236,"statfs":{"total":289910292480,"available":286739697664,"internally_reserved":0,"allocated":429051904,"data_stored":420453125,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299441,"internal_metadata":2741202511},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002736"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":517,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-10-11T11:19:02.772828+0000","last_change":"2023-10-11T11:15:31.661747+0000","last_active":"2023-10-11T11:19:02.772828+0000","last_peered":"2023-10-11T11:19:02.772828+0000","last_clean":"2023-10-11T11:15:27.638215+0000","last_became_active":"2023-10-11T11:15:31.661747+0000","last_became_peered":"2023-10-11T11:15:31.661747+0000","last_unstale":"2023-10-11T11:19:02.772828+0000","last_undegraded":"2023-10-11T11:19:02.772828+0000","last_fullsized":"2023-10-11T11:15:31.649336+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:15:25.047039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":28,"reported_epoch":499,"state":"active+undersized","last_fresh":"2023-10-11T11:19:00.628569+0000","last_change":"2023-10-11T11:15:31.671363+0000","last_active":"2023-10-11T11:19:00.628569+0000","last_peered":"2023-10-11T11:19:00.628569+0000","last_clean":"2023-10-11T10:43:40.751300+0000","last_became_active":"2023-10-11T11:15:31.671363+0000","last_became_peered":"2023-10-11T11:15:31.671363+0000","last_unstale":"2023-10-11T11:19:00.628569+0000","last_undegraded":"2023-10-11T11:19:00.628569+0000","last_fullsized":"2023-10-11T11:15:31.652001+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:01:53.884964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":512,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-10-11T11:19:03.429640+0000","last_change":"2023-10-11T10:43:41.772781+0000","last_active":"2023-10-11T11:19:03.429640+0000","last_peered":"2023-10-11T11:19:03.429640+0000","last_clean":"2023-10-11T11:19:03.429640+0000","last_became_active":"2023-10-11T10:43:41.772443+0000","last_became_peered":"2023-10-11T10:43:41.772443+0000","last_unstale":"2023-10-11T11:19:03.429640+0000","last_undegraded":"2023-10-11T11:19:03.429640+0000","last_fullsized":"2023-10-11T11:19:03.429640+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T14:15:30.547284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":513,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-10-11T11:19:02.772211+0000","last_change":"2023-10-11T10:43:41.772615+0000","last_active":"2023-10-11T11:19:02.772211+0000","last_peered":"2023-10-11T11:19:02.772211+0000","last_clean":"2023-10-11T11:19:02.772211+0000","last_became_active":"2023-10-11T10:43:41.772290+0000","last_became_peered":"2023-10-11T10:43:41.772290+0000","last_unstale":"2023-10-11T11:19:02.772211+0000","last_undegraded":"2023-10-11T11:19:02.772211+0000","last_fullsized":"2023-10-11T11:19:02.772211+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:26:06.909963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":647,"reported_epoch":499,"state":"active+undersized+degraded","last_fresh":"2023-10-11T11:19:00.625113+0000","last_change":"2023-10-11T11:15:31.654022+0000","last_active":"2023-10-11T11:19:00.625113+0000","last_peered":"2023-10-11T11:19:00.625113+0000","last_clean":"2023-10-11T11:15:27.636884+0000","last_became_active":"2023-10-11T11:15:31.654022+0000","last_became_peered":"2023-10-11T11:15:31.654022+0000","last_unstale":"2023-10-11T11:19:00.625113+0000","last_undegraded":"2023-10-11T11:15:31.646946+0000","last_fullsized":"2023-10-11T11:15:31.646751+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:36.739637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:36.739637+0000","last_clean_scrub_stamp":"2023-10-11T10:43:36.739637+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T11:33:33.696447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":519,"reported_epoch":499,"state":"active+clean+remapped","last_fresh":"2023-10-11T11:19:01.882635+0000","last_change":"2023-10-11T11:15:33.660200+0000","last_active":"2023-10-11T11:19:01.882635+0000","last_peered":"2023-10-11T11:19:01.882635+0000","last_clean":"2023-10-11T11:19:01.882635+0000","last_became_active":"2023-10-11T11:15:33.659619+0000","last_became_peered":"2023-10-11T11:15:33.659619+0000","last_unstale":"2023-10-11T11:19:01.882635+0000","last_undegraded":"2023-10-11T11:19:01.882635+0000","last_fullsized":"2023-10-11T11:19:01.882635+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T17:21:16.866392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":517,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-10-11T11:19:02.775128+0000","last_change":"2023-10-11T11:15:31.670813+0000","last_active":"2023-10-11T11:19:02.775128+0000","last_peered":"2023-10-11T11:19:02.775128+0000","last_clean":"2023-10-11T11:15:27.638219+0000","last_became_active":"2023-10-11T11:15:31.670813+0000","last_became_peered":"2023-10-11T11:15:31.670813+0000","last_unstale":"2023-10-11T11:19:02.775128+0000","last_undegraded":"2023-10-11T11:19:02.775128+0000","last_fullsized":"2023-10-11T11:15:31.658785+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:41:50.872346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":516,"reported_epoch":499,"state":"active+undersized","last_fresh":"2023-10-11T11:19:00.631668+0000","last_change":"2023-10-11T11:15:31.666378+0000","last_active":"2023-10-11T11:19:00.631668+0000","last_peered":"2023-10-11T11:19:00.631668+0000","last_clean":"2023-10-11T11:15:27.638303+0000","last_became_active":"2023-10-11T11:15:31.666378+0000","last_became_peered":"2023-10-11T11:15:31.666378+0000","last_unstale":"2023-10-11T11:19:00.631668+0000","last_undegraded":"2023-10-11T11:19:00.631668+0000","last_fullsized":"2023-10-11T11:15:31.649186+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T15:35:19.331209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":516,"reported_epoch":499,"state":"active+undersized","last_fresh":"2023-10-11T11:19:00.636960+0000","last_change":"2023-10-11T11:15:31.679465+0000","last_active":"2023-10-11T11:19:00.636960+0000","last_peered":"2023-10-11T11:19:00.636960+0000","last_clean":"2023-10-11T11:15:27.638749+0000","last_became_active":"2023-10-11T11:15:31.679465+0000","last_became_peered":"2023-10-11T11:15:31.679465+0000","last_unstale":"2023-10-11T11:19:00.636960+0000","last_undegraded":"2023-10-11T11:19:00.636960+0000","last_fullsized":"2023-10-11T11:15:31.659135+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T16:03:42.656692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":11,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738802,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":785348,"kb_used_data":5812,"kb_used_omap":96,"kb_used_meta":779423,"kb_avail":93586492,"statfs":{"total":96636764160,"available":95832567808,"internally_reserved":0,"allocated":5951488,"data_stored":3106918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":98609,"internal_metadata":798129871},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 11:14:37 2023","interfaces":[{"interface":"back","average":{"1min":0.314,"5min":0.316,"15min":0.443},"min":{"1min":0.24399999999999999,"5min":0.14799999999999999,"15min":0.14599999999999999},"max":{"1min":0.48499999999999999,"5min":1.712,"15min":9.6400000000000006},"last":0.31},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.35499999999999998,"15min":0.41299999999999998},"min":{"1min":0.20899999999999999,"5min":0.17199999999999999,"15min":0.16400000000000001},"max":{"1min":3.8180000000000001,"5min":3.8180000000000001,"15min":6.3200000000000003},"last":0.26500000000000001}]},{"osd":1,"last update":"Wed Oct 11 11:18:39 2023","interfaces":[{"interface":"back","average":{"1min":0.27500000000000002,"5min":0.29299999999999998,"15min":0.32400000000000001},"min":{"1min":0.154,"5min":0.154,"15min":0.154},"max":{"1min":0.41999999999999998,"5min":0.59299999999999997,"15min":2.9540000000000002},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.307,"5min":0.35899999999999999,"15min":0.36499999999999999},"min":{"1min":0.18099999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.45100000000000001,"5min":3.2719999999999998,"15min":3.2719999999999998},"last":0.45000000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738756,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1484496,"kb_used_data":406792,"kb_used_omap":107,"kb_used_meta":1077588,"kb_avail":92887344,"statfs":{"total":96636764160,"available":95116640256,"internally_reserved":0,"allocated":416555008,"data_stored":413648921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110309,"internal_metadata":1103450395},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Oct 11 11:14:30 2023","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.35399999999999998,"15min":0.40600000000000003},"min":{"1min":0.18099999999999999,"5min":0.155,"15min":0.14899999999999999},"max":{"1min":1.4319999999999999,"5min":2.3460000000000001,"15min":6.593},"last":0.26200000000000001},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.39100000000000001,"15min":0.48199999999999998},"min":{"1min":0.189,"5min":0.189,"15min":0.183},"max":{"1min":2.2120000000000002,"5min":2.4089999999999998,"15min":6.5960000000000001},"last":0.316}]},{"osd":2,"last update":"Wed Oct 11 11:14:30 2023","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.35499999999999998,"15min":0.47099999999999997},"min":{"1min":0.20300000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":1.2490000000000001,"5min":2.665,"15min":8.4730000000000008},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.375,"15min":0.442},"min":{"1min":0.23200000000000001,"5min":0.20499999999999999,"15min":0.19600000000000001},"max":{"1min":1.28,"5min":2.6230000000000002,"15min":7.0129999999999999},"last":0.33600000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738803,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":826440,"kb_used_data":6392,"kb_used_omap":88,"kb_used_meta":819943,"kb_avail":93545400,"statfs":{"total":96636764160,"available":95790489600,"internally_reserved":0,"allocated":6545408,"data_stored":3697286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":90523,"internal_metadata":839622245},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 11:14:33 2023","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.42799999999999999,"15min":0.41099999999999998},"min":{"1min":0.151,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":1.2809999999999999,"5min":3.5739999999999998,"15min":8.2479999999999993},"last":0.223},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.48299999999999998,"15min":0.49199999999999999},"min":{"1min":0.20699999999999999,"5min":0.182,"15min":0.17999999999999999},"max":{"1min":1.323,"5min":3.6080000000000001,"15min":8.2870000000000008},"last":0.28399999999999997}]},{"osd":2,"last update":"Wed Oct 11 11:18:40 2023","interfaces":[{"interface":"back","average":{"1min":0.33400000000000002,"5min":0.378,"15min":0.35599999999999998},"min":{"1min":0.16500000000000001,"5min":0.14699999999999999,"15min":0.14699999999999999},"max":{"1min":1.244,"5min":7.4829999999999997,"15min":10.617000000000001},"last":0.371},{"interface":"front","average":{"1min":0.34599999999999997,"5min":0.39000000000000001,"15min":0.34999999999999998},"min":{"1min":0.20399999999999999,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.81100000000000005,"5min":7.5350000000000001,"15min":7.5350000000000001},"last":0.41999999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-11T11:19:31.418 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-10-11T11:19:31.418 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 517, 'reported_epoch': 500, 'state': 'active+undersized', 'last_fresh': '2023-10-11T11:19:02.772828+0000', 'last_change': '2023-10-11T11:15:31.661747+0000', 'last_active': '2023-10-11T11:19:02.772828+0000', 'last_peered': '2023-10-11T11:19:02.772828+0000', 'last_clean': '2023-10-11T11:15:27.638215+0000', 'last_became_active': '2023-10-11T11:15:31.661747+0000', 'last_became_peered': '2023-10-11T11:15:31.661747+0000', 'last_unstale': '2023-10-11T11:19:02.772828+0000', 'last_undegraded': '2023-10-11T11:19:02.772828+0000', 'last_fullsized': '2023-10-11T11:15:31.649336+0000', 'mapping_epoch': 480, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_clean_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-12T22:15:25.047039+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-10-11T11:19:31.419 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-10-11T11:19:31.419 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 28, 'reported_epoch': 499, 'state': 'active+undersized', 'last_fresh': '2023-10-11T11:19:00.628569+0000', 'last_change': '2023-10-11T11:15:31.671363+0000', 'last_active': '2023-10-11T11:19:00.628569+0000', 'last_peered': '2023-10-11T11:19:00.628569+0000', 'last_clean': '2023-10-11T10:43:40.751300+0000', 'last_became_active': '2023-10-11T11:15:31.671363+0000', 'last_became_peered': '2023-10-11T11:15:31.671363+0000', 'last_unstale': '2023-10-11T11:19:00.628569+0000', 'last_undegraded': '2023-10-11T11:19:00.628569+0000', 'last_fullsized': '2023-10-11T11:15:31.652001+0000', 'mapping_epoch': 480, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_clean_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-12T21:01:53.884964+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-10-11T11:19:31.419 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-10-11T11:19:31.420 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "11'97", 'reported_seq': 647, 'reported_epoch': 499, 'state': 'active+undersized+degraded', 'last_fresh': '2023-10-11T11:19:00.625113+0000', 'last_change': '2023-10-11T11:15:31.654022+0000', 'last_active': '2023-10-11T11:19:00.625113+0000', 'last_peered': '2023-10-11T11:19:00.625113+0000', 'last_clean': '2023-10-11T11:15:27.636884+0000', 'last_became_active': '2023-10-11T11:15:31.654022+0000', 'last_became_peered': '2023-10-11T11:15:31.654022+0000', 'last_unstale': '2023-10-11T11:19:00.625113+0000', 'last_undegraded': '2023-10-11T11:15:31.646946+0000', 'last_fullsized': '2023-10-11T11:15:31.646751+0000', 'mapping_epoch': 480, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-11T10:43:36.739637+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-11T10:43:36.739637+0000', 'last_clean_scrub_stamp': '2023-10-11T10:43:36.739637+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-12T11:33:33.696447+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-10-11T11:19:31.420 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.0 is not active+clean 2023-10-11T11:19:31.420 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 519, 'reported_epoch': 499, 'state': 'active+clean+remapped', 'last_fresh': '2023-10-11T11:19:01.882635+0000', 'last_change': '2023-10-11T11:15:33.660200+0000', 'last_active': '2023-10-11T11:19:01.882635+0000', 'last_peered': '2023-10-11T11:19:01.882635+0000', 'last_clean': '2023-10-11T11:19:01.882635+0000', 'last_became_active': '2023-10-11T11:15:33.659619+0000', 'last_became_peered': '2023-10-11T11:15:33.659619+0000', 'last_unstale': '2023-10-11T11:19:01.882635+0000', 'last_undegraded': '2023-10-11T11:19:01.882635+0000', 'last_fullsized': '2023-10-11T11:19:01.882635+0000', 'mapping_epoch': 482, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 483, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_clean_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-12T17:21:16.866392+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2, 1], 'avail_no_missing': ['2', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-10-11T11:19:31.421 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-10-11T11:19:31.421 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 517, 'reported_epoch': 500, 'state': 'active+undersized', 'last_fresh': '2023-10-11T11:19:02.775128+0000', 'last_change': '2023-10-11T11:15:31.670813+0000', 'last_active': '2023-10-11T11:19:02.775128+0000', 'last_peered': '2023-10-11T11:19:02.775128+0000', 'last_clean': '2023-10-11T11:15:27.638219+0000', 'last_became_active': '2023-10-11T11:15:31.670813+0000', 'last_became_peered': '2023-10-11T11:15:31.670813+0000', 'last_unstale': '2023-10-11T11:19:02.775128+0000', 'last_undegraded': '2023-10-11T11:19:02.775128+0000', 'last_fullsized': '2023-10-11T11:15:31.658785+0000', 'mapping_epoch': 480, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_clean_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-12T21:41:50.872346+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-10-11T11:19:31.421 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-10-11T11:19:31.422 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 516, 'reported_epoch': 499, 'state': 'active+undersized', 'last_fresh': '2023-10-11T11:19:00.631668+0000', 'last_change': '2023-10-11T11:15:31.666378+0000', 'last_active': '2023-10-11T11:19:00.631668+0000', 'last_peered': '2023-10-11T11:19:00.631668+0000', 'last_clean': '2023-10-11T11:15:27.638303+0000', 'last_became_active': '2023-10-11T11:15:31.666378+0000', 'last_became_peered': '2023-10-11T11:15:31.666378+0000', 'last_unstale': '2023-10-11T11:19:00.631668+0000', 'last_undegraded': '2023-10-11T11:19:00.631668+0000', 'last_fullsized': '2023-10-11T11:15:31.649186+0000', 'mapping_epoch': 480, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_clean_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-12T15:35:19.331209+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-10-11T11:19:31.422 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-10-11T11:19:31.422 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 516, 'reported_epoch': 499, 'state': 'active+undersized', 'last_fresh': '2023-10-11T11:19:00.636960+0000', 'last_change': '2023-10-11T11:15:31.679465+0000', 'last_active': '2023-10-11T11:19:00.636960+0000', 'last_peered': '2023-10-11T11:19:00.636960+0000', 'last_clean': '2023-10-11T11:15:27.638749+0000', 'last_became_active': '2023-10-11T11:15:31.679465+0000', 'last_became_peered': '2023-10-11T11:15:31.679465+0000', 'last_unstale': '2023-10-11T11:19:00.636960+0000', 'last_undegraded': '2023-10-11T11:19:00.636960+0000', 'last_fullsized': '2023-10-11T11:15:31.659135+0000', 'mapping_epoch': 480, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_clean_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-12T16:03:42.656692+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-10-11T11:19:31.423 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-10-11T11:19:31.821 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T11:19:31.822 INFO:teuthology.orchestra.run.smithi111.stdout:{"fsid":"022ce6f2-9a5b-45ad-8b77-a1332d92f228","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2167,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":500,"num_osds":3,"num_up_osds":2,"osd_up_since":1697022930,"num_in_osds":3,"osd_in_since":1697021005,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":2},{"state_name":"active+undersized+degraded","count":1},{"state_name":"active+clean+remapped","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3170594816,"bytes_avail":286739697664,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":36,"modified":"2023-10-11T11:19:04.644408+0000","services":{}},"progress_events":{}} 2023-10-11T11:19:31.836 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-10-11T11:19:31.836 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-10-11T11:19:32.150 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T11:19:32.152 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2023-10-11T11:19:32.164 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":1576,"stamp":"2023-10-11T11:19:30.650916+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":11,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":107,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3096284,"kb_used_data":418996,"kb_used_omap":292,"kb_used_meta":2676955,"kb_avail":280019236,"statfs":{"total":289910292480,"available":286739697664,"internally_reserved":0,"allocated":429051904,"data_stored":420453125,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299441,"internal_metadata":2741202511},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002736"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":517,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-10-11T11:19:02.772828+0000","last_change":"2023-10-11T11:15:31.661747+0000","last_active":"2023-10-11T11:19:02.772828+0000","last_peered":"2023-10-11T11:19:02.772828+0000","last_clean":"2023-10-11T11:15:27.638215+0000","last_became_active":"2023-10-11T11:15:31.661747+0000","last_became_peered":"2023-10-11T11:15:31.661747+0000","last_unstale":"2023-10-11T11:19:02.772828+0000","last_undegraded":"2023-10-11T11:19:02.772828+0000","last_fullsized":"2023-10-11T11:15:31.649336+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:15:25.047039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":28,"reported_epoch":499,"state":"active+undersized","last_fresh":"2023-10-11T11:19:00.628569+0000","last_change":"2023-10-11T11:15:31.671363+0000","last_active":"2023-10-11T11:19:00.628569+0000","last_peered":"2023-10-11T11:19:00.628569+0000","last_clean":"2023-10-11T10:43:40.751300+0000","last_became_active":"2023-10-11T11:15:31.671363+0000","last_became_peered":"2023-10-11T11:15:31.671363+0000","last_unstale":"2023-10-11T11:19:00.628569+0000","last_undegraded":"2023-10-11T11:19:00.628569+0000","last_fullsized":"2023-10-11T11:15:31.652001+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:01:53.884964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":512,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-10-11T11:19:03.429640+0000","last_change":"2023-10-11T10:43:41.772781+0000","last_active":"2023-10-11T11:19:03.429640+0000","last_peered":"2023-10-11T11:19:03.429640+0000","last_clean":"2023-10-11T11:19:03.429640+0000","last_became_active":"2023-10-11T10:43:41.772443+0000","last_became_peered":"2023-10-11T10:43:41.772443+0000","last_unstale":"2023-10-11T11:19:03.429640+0000","last_undegraded":"2023-10-11T11:19:03.429640+0000","last_fullsized":"2023-10-11T11:19:03.429640+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T14:15:30.547284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":513,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-10-11T11:19:02.772211+0000","last_change":"2023-10-11T10:43:41.772615+0000","last_active":"2023-10-11T11:19:02.772211+0000","last_peered":"2023-10-11T11:19:02.772211+0000","last_clean":"2023-10-11T11:19:02.772211+0000","last_became_active":"2023-10-11T10:43:41.772290+0000","last_became_peered":"2023-10-11T10:43:41.772290+0000","last_unstale":"2023-10-11T11:19:02.772211+0000","last_undegraded":"2023-10-11T11:19:02.772211+0000","last_fullsized":"2023-10-11T11:19:02.772211+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:26:06.909963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":647,"reported_epoch":499,"state":"active+undersized+degraded","last_fresh":"2023-10-11T11:19:00.625113+0000","last_change":"2023-10-11T11:15:31.654022+0000","last_active":"2023-10-11T11:19:00.625113+0000","last_peered":"2023-10-11T11:19:00.625113+0000","last_clean":"2023-10-11T11:15:27.636884+0000","last_became_active":"2023-10-11T11:15:31.654022+0000","last_became_peered":"2023-10-11T11:15:31.654022+0000","last_unstale":"2023-10-11T11:19:00.625113+0000","last_undegraded":"2023-10-11T11:15:31.646946+0000","last_fullsized":"2023-10-11T11:15:31.646751+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:36.739637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:36.739637+0000","last_clean_scrub_stamp":"2023-10-11T10:43:36.739637+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T11:33:33.696447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":519,"reported_epoch":499,"state":"active+clean+remapped","last_fresh":"2023-10-11T11:19:01.882635+0000","last_change":"2023-10-11T11:15:33.660200+0000","last_active":"2023-10-11T11:19:01.882635+0000","last_peered":"2023-10-11T11:19:01.882635+0000","last_clean":"2023-10-11T11:19:01.882635+0000","last_became_active":"2023-10-11T11:15:33.659619+0000","last_became_peered":"2023-10-11T11:15:33.659619+0000","last_unstale":"2023-10-11T11:19:01.882635+0000","last_undegraded":"2023-10-11T11:19:01.882635+0000","last_fullsized":"2023-10-11T11:19:01.882635+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T17:21:16.866392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":517,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-10-11T11:19:02.775128+0000","last_change":"2023-10-11T11:15:31.670813+0000","last_active":"2023-10-11T11:19:02.775128+0000","last_peered":"2023-10-11T11:19:02.775128+0000","last_clean":"2023-10-11T11:15:27.638219+0000","last_became_active":"2023-10-11T11:15:31.670813+0000","last_became_peered":"2023-10-11T11:15:31.670813+0000","last_unstale":"2023-10-11T11:19:02.775128+0000","last_undegraded":"2023-10-11T11:19:02.775128+0000","last_fullsized":"2023-10-11T11:15:31.658785+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:41:50.872346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":516,"reported_epoch":499,"state":"active+undersized","last_fresh":"2023-10-11T11:19:00.631668+0000","last_change":"2023-10-11T11:15:31.666378+0000","last_active":"2023-10-11T11:19:00.631668+0000","last_peered":"2023-10-11T11:19:00.631668+0000","last_clean":"2023-10-11T11:15:27.638303+0000","last_became_active":"2023-10-11T11:15:31.666378+0000","last_became_peered":"2023-10-11T11:15:31.666378+0000","last_unstale":"2023-10-11T11:19:00.631668+0000","last_undegraded":"2023-10-11T11:19:00.631668+0000","last_fullsized":"2023-10-11T11:15:31.649186+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T15:35:19.331209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":516,"reported_epoch":499,"state":"active+undersized","last_fresh":"2023-10-11T11:19:00.636960+0000","last_change":"2023-10-11T11:15:31.679465+0000","last_active":"2023-10-11T11:19:00.636960+0000","last_peered":"2023-10-11T11:19:00.636960+0000","last_clean":"2023-10-11T11:15:27.638749+0000","last_became_active":"2023-10-11T11:15:31.679465+0000","last_became_peered":"2023-10-11T11:15:31.679465+0000","last_unstale":"2023-10-11T11:19:00.636960+0000","last_undegraded":"2023-10-11T11:19:00.636960+0000","last_fullsized":"2023-10-11T11:15:31.659135+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T16:03:42.656692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":11,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738802,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":785348,"kb_used_data":5812,"kb_used_omap":96,"kb_used_meta":779423,"kb_avail":93586492,"statfs":{"total":96636764160,"available":95832567808,"internally_reserved":0,"allocated":5951488,"data_stored":3106918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":98609,"internal_metadata":798129871},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 11:14:37 2023","interfaces":[{"interface":"back","average":{"1min":0.314,"5min":0.316,"15min":0.443},"min":{"1min":0.24399999999999999,"5min":0.14799999999999999,"15min":0.14599999999999999},"max":{"1min":0.48499999999999999,"5min":1.712,"15min":9.6400000000000006},"last":0.31},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.35499999999999998,"15min":0.41299999999999998},"min":{"1min":0.20899999999999999,"5min":0.17199999999999999,"15min":0.16400000000000001},"max":{"1min":3.8180000000000001,"5min":3.8180000000000001,"15min":6.3200000000000003},"last":0.26500000000000001}]},{"osd":1,"last update":"Wed Oct 11 11:18:39 2023","interfaces":[{"interface":"back","average":{"1min":0.27500000000000002,"5min":0.29299999999999998,"15min":0.32400000000000001},"min":{"1min":0.154,"5min":0.154,"15min":0.154},"max":{"1min":0.41999999999999998,"5min":0.59299999999999997,"15min":2.9540000000000002},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.307,"5min":0.35899999999999999,"15min":0.36499999999999999},"min":{"1min":0.18099999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.45100000000000001,"5min":3.2719999999999998,"15min":3.2719999999999998},"last":0.45000000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738756,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1484496,"kb_used_data":406792,"kb_used_omap":107,"kb_used_meta":1077588,"kb_avail":92887344,"statfs":{"total":96636764160,"available":95116640256,"internally_reserved":0,"allocated":416555008,"data_stored":413648921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110309,"internal_metadata":1103450395},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Oct 11 11:14:30 2023","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.35399999999999998,"15min":0.40600000000000003},"min":{"1min":0.18099999999999999,"5min":0.155,"15min":0.14899999999999999},"max":{"1min":1.4319999999999999,"5min":2.3460000000000001,"15min":6.593},"last":0.26200000000000001},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.39100000000000001,"15min":0.48199999999999998},"min":{"1min":0.189,"5min":0.189,"15min":0.183},"max":{"1min":2.2120000000000002,"5min":2.4089999999999998,"15min":6.5960000000000001},"last":0.316}]},{"osd":2,"last update":"Wed Oct 11 11:14:30 2023","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.35499999999999998,"15min":0.47099999999999997},"min":{"1min":0.20300000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":1.2490000000000001,"5min":2.665,"15min":8.4730000000000008},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.375,"15min":0.442},"min":{"1min":0.23200000000000001,"5min":0.20499999999999999,"15min":0.19600000000000001},"max":{"1min":1.28,"5min":2.6230000000000002,"15min":7.0129999999999999},"last":0.33600000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738803,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":826440,"kb_used_data":6392,"kb_used_omap":88,"kb_used_meta":819943,"kb_avail":93545400,"statfs":{"total":96636764160,"available":95790489600,"internally_reserved":0,"allocated":6545408,"data_stored":3697286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":90523,"internal_metadata":839622245},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 11:14:33 2023","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.42799999999999999,"15min":0.41099999999999998},"min":{"1min":0.151,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":1.2809999999999999,"5min":3.5739999999999998,"15min":8.2479999999999993},"last":0.223},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.48299999999999998,"15min":0.49199999999999999},"min":{"1min":0.20699999999999999,"5min":0.182,"15min":0.17999999999999999},"max":{"1min":1.323,"5min":3.6080000000000001,"15min":8.2870000000000008},"last":0.28399999999999997}]},{"osd":2,"last update":"Wed Oct 11 11:18:40 2023","interfaces":[{"interface":"back","average":{"1min":0.33400000000000002,"5min":0.378,"15min":0.35599999999999998},"min":{"1min":0.16500000000000001,"5min":0.14699999999999999,"15min":0.14699999999999999},"max":{"1min":1.244,"5min":7.4829999999999997,"15min":10.617000000000001},"last":0.371},{"interface":"front","average":{"1min":0.34599999999999997,"5min":0.39000000000000001,"15min":0.34999999999999998},"min":{"1min":0.20399999999999999,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.81100000000000005,"5min":7.5350000000000001,"15min":7.5350000000000001},"last":0.41999999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-11T11:19:35.166 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-10-11T11:19:35.331 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~243s 2023-10-11T11:19:35.497 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T11:19:35.499 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2023-10-11T11:19:35.510 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":1578,"stamp":"2023-10-11T11:19:34.651828+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":11,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":107,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3096284,"kb_used_data":418996,"kb_used_omap":292,"kb_used_meta":2676955,"kb_avail":280019236,"statfs":{"total":289910292480,"available":286739697664,"internally_reserved":0,"allocated":429051904,"data_stored":420453125,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299441,"internal_metadata":2741202511},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002857"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":517,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-10-11T11:19:02.772828+0000","last_change":"2023-10-11T11:15:31.661747+0000","last_active":"2023-10-11T11:19:02.772828+0000","last_peered":"2023-10-11T11:19:02.772828+0000","last_clean":"2023-10-11T11:15:27.638215+0000","last_became_active":"2023-10-11T11:15:31.661747+0000","last_became_peered":"2023-10-11T11:15:31.661747+0000","last_unstale":"2023-10-11T11:19:02.772828+0000","last_undegraded":"2023-10-11T11:19:02.772828+0000","last_fullsized":"2023-10-11T11:15:31.649336+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:15:25.047039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":28,"reported_epoch":499,"state":"active+undersized","last_fresh":"2023-10-11T11:19:00.628569+0000","last_change":"2023-10-11T11:15:31.671363+0000","last_active":"2023-10-11T11:19:00.628569+0000","last_peered":"2023-10-11T11:19:00.628569+0000","last_clean":"2023-10-11T10:43:40.751300+0000","last_became_active":"2023-10-11T11:15:31.671363+0000","last_became_peered":"2023-10-11T11:15:31.671363+0000","last_unstale":"2023-10-11T11:19:00.628569+0000","last_undegraded":"2023-10-11T11:19:00.628569+0000","last_fullsized":"2023-10-11T11:15:31.652001+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:01:53.884964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":512,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-10-11T11:19:03.429640+0000","last_change":"2023-10-11T10:43:41.772781+0000","last_active":"2023-10-11T11:19:03.429640+0000","last_peered":"2023-10-11T11:19:03.429640+0000","last_clean":"2023-10-11T11:19:03.429640+0000","last_became_active":"2023-10-11T10:43:41.772443+0000","last_became_peered":"2023-10-11T10:43:41.772443+0000","last_unstale":"2023-10-11T11:19:03.429640+0000","last_undegraded":"2023-10-11T11:19:03.429640+0000","last_fullsized":"2023-10-11T11:19:03.429640+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T14:15:30.547284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":513,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-10-11T11:19:02.772211+0000","last_change":"2023-10-11T10:43:41.772615+0000","last_active":"2023-10-11T11:19:02.772211+0000","last_peered":"2023-10-11T11:19:02.772211+0000","last_clean":"2023-10-11T11:19:02.772211+0000","last_became_active":"2023-10-11T10:43:41.772290+0000","last_became_peered":"2023-10-11T10:43:41.772290+0000","last_unstale":"2023-10-11T11:19:02.772211+0000","last_undegraded":"2023-10-11T11:19:02.772211+0000","last_fullsized":"2023-10-11T11:19:02.772211+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:26:06.909963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":647,"reported_epoch":499,"state":"active+undersized+degraded","last_fresh":"2023-10-11T11:19:00.625113+0000","last_change":"2023-10-11T11:15:31.654022+0000","last_active":"2023-10-11T11:19:00.625113+0000","last_peered":"2023-10-11T11:19:00.625113+0000","last_clean":"2023-10-11T11:15:27.636884+0000","last_became_active":"2023-10-11T11:15:31.654022+0000","last_became_peered":"2023-10-11T11:15:31.654022+0000","last_unstale":"2023-10-11T11:19:00.625113+0000","last_undegraded":"2023-10-11T11:15:31.646946+0000","last_fullsized":"2023-10-11T11:15:31.646751+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:36.739637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:36.739637+0000","last_clean_scrub_stamp":"2023-10-11T10:43:36.739637+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T11:33:33.696447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":519,"reported_epoch":499,"state":"active+clean+remapped","last_fresh":"2023-10-11T11:19:01.882635+0000","last_change":"2023-10-11T11:15:33.660200+0000","last_active":"2023-10-11T11:19:01.882635+0000","last_peered":"2023-10-11T11:19:01.882635+0000","last_clean":"2023-10-11T11:19:01.882635+0000","last_became_active":"2023-10-11T11:15:33.659619+0000","last_became_peered":"2023-10-11T11:15:33.659619+0000","last_unstale":"2023-10-11T11:19:01.882635+0000","last_undegraded":"2023-10-11T11:19:01.882635+0000","last_fullsized":"2023-10-11T11:19:01.882635+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T17:21:16.866392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":517,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-10-11T11:19:02.775128+0000","last_change":"2023-10-11T11:15:31.670813+0000","last_active":"2023-10-11T11:19:02.775128+0000","last_peered":"2023-10-11T11:19:02.775128+0000","last_clean":"2023-10-11T11:15:27.638219+0000","last_became_active":"2023-10-11T11:15:31.670813+0000","last_became_peered":"2023-10-11T11:15:31.670813+0000","last_unstale":"2023-10-11T11:19:02.775128+0000","last_undegraded":"2023-10-11T11:19:02.775128+0000","last_fullsized":"2023-10-11T11:15:31.658785+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:41:50.872346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":516,"reported_epoch":499,"state":"active+undersized","last_fresh":"2023-10-11T11:19:00.631668+0000","last_change":"2023-10-11T11:15:31.666378+0000","last_active":"2023-10-11T11:19:00.631668+0000","last_peered":"2023-10-11T11:19:00.631668+0000","last_clean":"2023-10-11T11:15:27.638303+0000","last_became_active":"2023-10-11T11:15:31.666378+0000","last_became_peered":"2023-10-11T11:15:31.666378+0000","last_unstale":"2023-10-11T11:19:00.631668+0000","last_undegraded":"2023-10-11T11:19:00.631668+0000","last_fullsized":"2023-10-11T11:15:31.649186+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T15:35:19.331209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":516,"reported_epoch":499,"state":"active+undersized","last_fresh":"2023-10-11T11:19:00.636960+0000","last_change":"2023-10-11T11:15:31.679465+0000","last_active":"2023-10-11T11:19:00.636960+0000","last_peered":"2023-10-11T11:19:00.636960+0000","last_clean":"2023-10-11T11:15:27.638749+0000","last_became_active":"2023-10-11T11:15:31.679465+0000","last_became_peered":"2023-10-11T11:15:31.679465+0000","last_unstale":"2023-10-11T11:19:00.636960+0000","last_undegraded":"2023-10-11T11:19:00.636960+0000","last_fullsized":"2023-10-11T11:15:31.659135+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T16:03:42.656692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":11,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738802,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":785348,"kb_used_data":5812,"kb_used_omap":96,"kb_used_meta":779423,"kb_avail":93586492,"statfs":{"total":96636764160,"available":95832567808,"internally_reserved":0,"allocated":5951488,"data_stored":3106918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":98609,"internal_metadata":798129871},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 11:14:37 2023","interfaces":[{"interface":"back","average":{"1min":0.314,"5min":0.316,"15min":0.443},"min":{"1min":0.24399999999999999,"5min":0.14799999999999999,"15min":0.14599999999999999},"max":{"1min":0.48499999999999999,"5min":1.712,"15min":9.6400000000000006},"last":0.31},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.35499999999999998,"15min":0.41299999999999998},"min":{"1min":0.20899999999999999,"5min":0.17199999999999999,"15min":0.16400000000000001},"max":{"1min":3.8180000000000001,"5min":3.8180000000000001,"15min":6.3200000000000003},"last":0.26500000000000001}]},{"osd":1,"last update":"Wed Oct 11 11:18:39 2023","interfaces":[{"interface":"back","average":{"1min":0.27500000000000002,"5min":0.29299999999999998,"15min":0.32400000000000001},"min":{"1min":0.154,"5min":0.154,"15min":0.154},"max":{"1min":0.41999999999999998,"5min":0.59299999999999997,"15min":2.9540000000000002},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.307,"5min":0.35899999999999999,"15min":0.36499999999999999},"min":{"1min":0.18099999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.45100000000000001,"5min":3.2719999999999998,"15min":3.2719999999999998},"last":0.45000000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738756,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1484496,"kb_used_data":406792,"kb_used_omap":107,"kb_used_meta":1077588,"kb_avail":92887344,"statfs":{"total":96636764160,"available":95116640256,"internally_reserved":0,"allocated":416555008,"data_stored":413648921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110309,"internal_metadata":1103450395},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Oct 11 11:14:30 2023","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.35399999999999998,"15min":0.40600000000000003},"min":{"1min":0.18099999999999999,"5min":0.155,"15min":0.14899999999999999},"max":{"1min":1.4319999999999999,"5min":2.3460000000000001,"15min":6.593},"last":0.26200000000000001},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.39100000000000001,"15min":0.48199999999999998},"min":{"1min":0.189,"5min":0.189,"15min":0.183},"max":{"1min":2.2120000000000002,"5min":2.4089999999999998,"15min":6.5960000000000001},"last":0.316}]},{"osd":2,"last update":"Wed Oct 11 11:14:30 2023","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.35499999999999998,"15min":0.47099999999999997},"min":{"1min":0.20300000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":1.2490000000000001,"5min":2.665,"15min":8.4730000000000008},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.375,"15min":0.442},"min":{"1min":0.23200000000000001,"5min":0.20499999999999999,"15min":0.19600000000000001},"max":{"1min":1.28,"5min":2.6230000000000002,"15min":7.0129999999999999},"last":0.33600000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738803,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":826440,"kb_used_data":6392,"kb_used_omap":88,"kb_used_meta":819943,"kb_avail":93545400,"statfs":{"total":96636764160,"available":95790489600,"internally_reserved":0,"allocated":6545408,"data_stored":3697286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":90523,"internal_metadata":839622245},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 11:14:33 2023","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.42799999999999999,"15min":0.41099999999999998},"min":{"1min":0.151,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":1.2809999999999999,"5min":3.5739999999999998,"15min":8.2479999999999993},"last":0.223},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.48299999999999998,"15min":0.49199999999999999},"min":{"1min":0.20699999999999999,"5min":0.182,"15min":0.17999999999999999},"max":{"1min":1.323,"5min":3.6080000000000001,"15min":8.2870000000000008},"last":0.28399999999999997}]},{"osd":2,"last update":"Wed Oct 11 11:18:40 2023","interfaces":[{"interface":"back","average":{"1min":0.33400000000000002,"5min":0.378,"15min":0.35599999999999998},"min":{"1min":0.16500000000000001,"5min":0.14699999999999999,"15min":0.14699999999999999},"max":{"1min":1.244,"5min":7.4829999999999997,"15min":10.617000000000001},"last":0.371},{"interface":"front","average":{"1min":0.34599999999999997,"5min":0.39000000000000001,"15min":0.34999999999999998},"min":{"1min":0.20399999999999999,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.81100000000000005,"5min":7.5350000000000001,"15min":7.5350000000000001},"last":0.41999999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-11T11:19:35.512 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-10-11T11:19:35.831 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T11:19:35.833 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2023-10-11T11:19:35.844 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":1578,"stamp":"2023-10-11T11:19:34.651828+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":11,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":107,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3096284,"kb_used_data":418996,"kb_used_omap":292,"kb_used_meta":2676955,"kb_avail":280019236,"statfs":{"total":289910292480,"available":286739697664,"internally_reserved":0,"allocated":429051904,"data_stored":420453125,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299441,"internal_metadata":2741202511},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002857"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":517,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-10-11T11:19:02.772828+0000","last_change":"2023-10-11T11:15:31.661747+0000","last_active":"2023-10-11T11:19:02.772828+0000","last_peered":"2023-10-11T11:19:02.772828+0000","last_clean":"2023-10-11T11:15:27.638215+0000","last_became_active":"2023-10-11T11:15:31.661747+0000","last_became_peered":"2023-10-11T11:15:31.661747+0000","last_unstale":"2023-10-11T11:19:02.772828+0000","last_undegraded":"2023-10-11T11:19:02.772828+0000","last_fullsized":"2023-10-11T11:15:31.649336+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:15:25.047039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":28,"reported_epoch":499,"state":"active+undersized","last_fresh":"2023-10-11T11:19:00.628569+0000","last_change":"2023-10-11T11:15:31.671363+0000","last_active":"2023-10-11T11:19:00.628569+0000","last_peered":"2023-10-11T11:19:00.628569+0000","last_clean":"2023-10-11T10:43:40.751300+0000","last_became_active":"2023-10-11T11:15:31.671363+0000","last_became_peered":"2023-10-11T11:15:31.671363+0000","last_unstale":"2023-10-11T11:19:00.628569+0000","last_undegraded":"2023-10-11T11:19:00.628569+0000","last_fullsized":"2023-10-11T11:15:31.652001+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:01:53.884964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":512,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-10-11T11:19:03.429640+0000","last_change":"2023-10-11T10:43:41.772781+0000","last_active":"2023-10-11T11:19:03.429640+0000","last_peered":"2023-10-11T11:19:03.429640+0000","last_clean":"2023-10-11T11:19:03.429640+0000","last_became_active":"2023-10-11T10:43:41.772443+0000","last_became_peered":"2023-10-11T10:43:41.772443+0000","last_unstale":"2023-10-11T11:19:03.429640+0000","last_undegraded":"2023-10-11T11:19:03.429640+0000","last_fullsized":"2023-10-11T11:19:03.429640+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T14:15:30.547284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":513,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-10-11T11:19:02.772211+0000","last_change":"2023-10-11T10:43:41.772615+0000","last_active":"2023-10-11T11:19:02.772211+0000","last_peered":"2023-10-11T11:19:02.772211+0000","last_clean":"2023-10-11T11:19:02.772211+0000","last_became_active":"2023-10-11T10:43:41.772290+0000","last_became_peered":"2023-10-11T10:43:41.772290+0000","last_unstale":"2023-10-11T11:19:02.772211+0000","last_undegraded":"2023-10-11T11:19:02.772211+0000","last_fullsized":"2023-10-11T11:19:02.772211+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:26:06.909963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":647,"reported_epoch":499,"state":"active+undersized+degraded","last_fresh":"2023-10-11T11:19:00.625113+0000","last_change":"2023-10-11T11:15:31.654022+0000","last_active":"2023-10-11T11:19:00.625113+0000","last_peered":"2023-10-11T11:19:00.625113+0000","last_clean":"2023-10-11T11:15:27.636884+0000","last_became_active":"2023-10-11T11:15:31.654022+0000","last_became_peered":"2023-10-11T11:15:31.654022+0000","last_unstale":"2023-10-11T11:19:00.625113+0000","last_undegraded":"2023-10-11T11:15:31.646946+0000","last_fullsized":"2023-10-11T11:15:31.646751+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:36.739637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:36.739637+0000","last_clean_scrub_stamp":"2023-10-11T10:43:36.739637+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T11:33:33.696447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":519,"reported_epoch":499,"state":"active+clean+remapped","last_fresh":"2023-10-11T11:19:01.882635+0000","last_change":"2023-10-11T11:15:33.660200+0000","last_active":"2023-10-11T11:19:01.882635+0000","last_peered":"2023-10-11T11:19:01.882635+0000","last_clean":"2023-10-11T11:19:01.882635+0000","last_became_active":"2023-10-11T11:15:33.659619+0000","last_became_peered":"2023-10-11T11:15:33.659619+0000","last_unstale":"2023-10-11T11:19:01.882635+0000","last_undegraded":"2023-10-11T11:19:01.882635+0000","last_fullsized":"2023-10-11T11:19:01.882635+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T17:21:16.866392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":517,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-10-11T11:19:02.775128+0000","last_change":"2023-10-11T11:15:31.670813+0000","last_active":"2023-10-11T11:19:02.775128+0000","last_peered":"2023-10-11T11:19:02.775128+0000","last_clean":"2023-10-11T11:15:27.638219+0000","last_became_active":"2023-10-11T11:15:31.670813+0000","last_became_peered":"2023-10-11T11:15:31.670813+0000","last_unstale":"2023-10-11T11:19:02.775128+0000","last_undegraded":"2023-10-11T11:19:02.775128+0000","last_fullsized":"2023-10-11T11:15:31.658785+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:41:50.872346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":516,"reported_epoch":499,"state":"active+undersized","last_fresh":"2023-10-11T11:19:00.631668+0000","last_change":"2023-10-11T11:15:31.666378+0000","last_active":"2023-10-11T11:19:00.631668+0000","last_peered":"2023-10-11T11:19:00.631668+0000","last_clean":"2023-10-11T11:15:27.638303+0000","last_became_active":"2023-10-11T11:15:31.666378+0000","last_became_peered":"2023-10-11T11:15:31.666378+0000","last_unstale":"2023-10-11T11:19:00.631668+0000","last_undegraded":"2023-10-11T11:19:00.631668+0000","last_fullsized":"2023-10-11T11:15:31.649186+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T15:35:19.331209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":516,"reported_epoch":499,"state":"active+undersized","last_fresh":"2023-10-11T11:19:00.636960+0000","last_change":"2023-10-11T11:15:31.679465+0000","last_active":"2023-10-11T11:19:00.636960+0000","last_peered":"2023-10-11T11:19:00.636960+0000","last_clean":"2023-10-11T11:15:27.638749+0000","last_became_active":"2023-10-11T11:15:31.679465+0000","last_became_peered":"2023-10-11T11:15:31.679465+0000","last_unstale":"2023-10-11T11:19:00.636960+0000","last_undegraded":"2023-10-11T11:19:00.636960+0000","last_fullsized":"2023-10-11T11:15:31.659135+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T16:03:42.656692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":11,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738802,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":785348,"kb_used_data":5812,"kb_used_omap":96,"kb_used_meta":779423,"kb_avail":93586492,"statfs":{"total":96636764160,"available":95832567808,"internally_reserved":0,"allocated":5951488,"data_stored":3106918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":98609,"internal_metadata":798129871},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 11:14:37 2023","interfaces":[{"interface":"back","average":{"1min":0.314,"5min":0.316,"15min":0.443},"min":{"1min":0.24399999999999999,"5min":0.14799999999999999,"15min":0.14599999999999999},"max":{"1min":0.48499999999999999,"5min":1.712,"15min":9.6400000000000006},"last":0.31},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.35499999999999998,"15min":0.41299999999999998},"min":{"1min":0.20899999999999999,"5min":0.17199999999999999,"15min":0.16400000000000001},"max":{"1min":3.8180000000000001,"5min":3.8180000000000001,"15min":6.3200000000000003},"last":0.26500000000000001}]},{"osd":1,"last update":"Wed Oct 11 11:18:39 2023","interfaces":[{"interface":"back","average":{"1min":0.27500000000000002,"5min":0.29299999999999998,"15min":0.32400000000000001},"min":{"1min":0.154,"5min":0.154,"15min":0.154},"max":{"1min":0.41999999999999998,"5min":0.59299999999999997,"15min":2.9540000000000002},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.307,"5min":0.35899999999999999,"15min":0.36499999999999999},"min":{"1min":0.18099999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.45100000000000001,"5min":3.2719999999999998,"15min":3.2719999999999998},"last":0.45000000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738756,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1484496,"kb_used_data":406792,"kb_used_omap":107,"kb_used_meta":1077588,"kb_avail":92887344,"statfs":{"total":96636764160,"available":95116640256,"internally_reserved":0,"allocated":416555008,"data_stored":413648921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110309,"internal_metadata":1103450395},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Oct 11 11:14:30 2023","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.35399999999999998,"15min":0.40600000000000003},"min":{"1min":0.18099999999999999,"5min":0.155,"15min":0.14899999999999999},"max":{"1min":1.4319999999999999,"5min":2.3460000000000001,"15min":6.593},"last":0.26200000000000001},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.39100000000000001,"15min":0.48199999999999998},"min":{"1min":0.189,"5min":0.189,"15min":0.183},"max":{"1min":2.2120000000000002,"5min":2.4089999999999998,"15min":6.5960000000000001},"last":0.316}]},{"osd":2,"last update":"Wed Oct 11 11:14:30 2023","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.35499999999999998,"15min":0.47099999999999997},"min":{"1min":0.20300000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":1.2490000000000001,"5min":2.665,"15min":8.4730000000000008},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.375,"15min":0.442},"min":{"1min":0.23200000000000001,"5min":0.20499999999999999,"15min":0.19600000000000001},"max":{"1min":1.28,"5min":2.6230000000000002,"15min":7.0129999999999999},"last":0.33600000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738803,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":826440,"kb_used_data":6392,"kb_used_omap":88,"kb_used_meta":819943,"kb_avail":93545400,"statfs":{"total":96636764160,"available":95790489600,"internally_reserved":0,"allocated":6545408,"data_stored":3697286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":90523,"internal_metadata":839622245},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 11:14:33 2023","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.42799999999999999,"15min":0.41099999999999998},"min":{"1min":0.151,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":1.2809999999999999,"5min":3.5739999999999998,"15min":8.2479999999999993},"last":0.223},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.48299999999999998,"15min":0.49199999999999999},"min":{"1min":0.20699999999999999,"5min":0.182,"15min":0.17999999999999999},"max":{"1min":1.323,"5min":3.6080000000000001,"15min":8.2870000000000008},"last":0.28399999999999997}]},{"osd":2,"last update":"Wed Oct 11 11:18:40 2023","interfaces":[{"interface":"back","average":{"1min":0.33400000000000002,"5min":0.378,"15min":0.35599999999999998},"min":{"1min":0.16500000000000001,"5min":0.14699999999999999,"15min":0.14699999999999999},"max":{"1min":1.244,"5min":7.4829999999999997,"15min":10.617000000000001},"last":0.371},{"interface":"front","average":{"1min":0.34599999999999997,"5min":0.39000000000000001,"15min":0.34999999999999998},"min":{"1min":0.20399999999999999,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.81100000000000005,"5min":7.5350000000000001,"15min":7.5350000000000001},"last":0.41999999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-11T11:19:35.846 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-10-11T11:19:35.846 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 517, 'reported_epoch': 500, 'state': 'active+undersized', 'last_fresh': '2023-10-11T11:19:02.772828+0000', 'last_change': '2023-10-11T11:15:31.661747+0000', 'last_active': '2023-10-11T11:19:02.772828+0000', 'last_peered': '2023-10-11T11:19:02.772828+0000', 'last_clean': '2023-10-11T11:15:27.638215+0000', 'last_became_active': '2023-10-11T11:15:31.661747+0000', 'last_became_peered': '2023-10-11T11:15:31.661747+0000', 'last_unstale': '2023-10-11T11:19:02.772828+0000', 'last_undegraded': '2023-10-11T11:19:02.772828+0000', 'last_fullsized': '2023-10-11T11:15:31.649336+0000', 'mapping_epoch': 480, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_clean_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-12T22:15:25.047039+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-10-11T11:19:35.846 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-10-11T11:19:35.847 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 28, 'reported_epoch': 499, 'state': 'active+undersized', 'last_fresh': '2023-10-11T11:19:00.628569+0000', 'last_change': '2023-10-11T11:15:31.671363+0000', 'last_active': '2023-10-11T11:19:00.628569+0000', 'last_peered': '2023-10-11T11:19:00.628569+0000', 'last_clean': '2023-10-11T10:43:40.751300+0000', 'last_became_active': '2023-10-11T11:15:31.671363+0000', 'last_became_peered': '2023-10-11T11:15:31.671363+0000', 'last_unstale': '2023-10-11T11:19:00.628569+0000', 'last_undegraded': '2023-10-11T11:19:00.628569+0000', 'last_fullsized': '2023-10-11T11:15:31.652001+0000', 'mapping_epoch': 480, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_clean_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-12T21:01:53.884964+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-10-11T11:19:35.847 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-10-11T11:19:35.847 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "11'97", 'reported_seq': 647, 'reported_epoch': 499, 'state': 'active+undersized+degraded', 'last_fresh': '2023-10-11T11:19:00.625113+0000', 'last_change': '2023-10-11T11:15:31.654022+0000', 'last_active': '2023-10-11T11:19:00.625113+0000', 'last_peered': '2023-10-11T11:19:00.625113+0000', 'last_clean': '2023-10-11T11:15:27.636884+0000', 'last_became_active': '2023-10-11T11:15:31.654022+0000', 'last_became_peered': '2023-10-11T11:15:31.654022+0000', 'last_unstale': '2023-10-11T11:19:00.625113+0000', 'last_undegraded': '2023-10-11T11:15:31.646946+0000', 'last_fullsized': '2023-10-11T11:15:31.646751+0000', 'mapping_epoch': 480, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-11T10:43:36.739637+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-11T10:43:36.739637+0000', 'last_clean_scrub_stamp': '2023-10-11T10:43:36.739637+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-12T11:33:33.696447+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-10-11T11:19:35.848 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.0 is not active+clean 2023-10-11T11:19:35.848 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 519, 'reported_epoch': 499, 'state': 'active+clean+remapped', 'last_fresh': '2023-10-11T11:19:01.882635+0000', 'last_change': '2023-10-11T11:15:33.660200+0000', 'last_active': '2023-10-11T11:19:01.882635+0000', 'last_peered': '2023-10-11T11:19:01.882635+0000', 'last_clean': '2023-10-11T11:19:01.882635+0000', 'last_became_active': '2023-10-11T11:15:33.659619+0000', 'last_became_peered': '2023-10-11T11:15:33.659619+0000', 'last_unstale': '2023-10-11T11:19:01.882635+0000', 'last_undegraded': '2023-10-11T11:19:01.882635+0000', 'last_fullsized': '2023-10-11T11:19:01.882635+0000', 'mapping_epoch': 482, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 483, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_clean_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-12T17:21:16.866392+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2, 1], 'avail_no_missing': ['2', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-10-11T11:19:35.848 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-10-11T11:19:35.849 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 517, 'reported_epoch': 500, 'state': 'active+undersized', 'last_fresh': '2023-10-11T11:19:02.775128+0000', 'last_change': '2023-10-11T11:15:31.670813+0000', 'last_active': '2023-10-11T11:19:02.775128+0000', 'last_peered': '2023-10-11T11:19:02.775128+0000', 'last_clean': '2023-10-11T11:15:27.638219+0000', 'last_became_active': '2023-10-11T11:15:31.670813+0000', 'last_became_peered': '2023-10-11T11:15:31.670813+0000', 'last_unstale': '2023-10-11T11:19:02.775128+0000', 'last_undegraded': '2023-10-11T11:19:02.775128+0000', 'last_fullsized': '2023-10-11T11:15:31.658785+0000', 'mapping_epoch': 480, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_clean_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-12T21:41:50.872346+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-10-11T11:19:35.849 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-10-11T11:19:35.849 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 516, 'reported_epoch': 499, 'state': 'active+undersized', 'last_fresh': '2023-10-11T11:19:00.631668+0000', 'last_change': '2023-10-11T11:15:31.666378+0000', 'last_active': '2023-10-11T11:19:00.631668+0000', 'last_peered': '2023-10-11T11:19:00.631668+0000', 'last_clean': '2023-10-11T11:15:27.638303+0000', 'last_became_active': '2023-10-11T11:15:31.666378+0000', 'last_became_peered': '2023-10-11T11:15:31.666378+0000', 'last_unstale': '2023-10-11T11:19:00.631668+0000', 'last_undegraded': '2023-10-11T11:19:00.631668+0000', 'last_fullsized': '2023-10-11T11:15:31.649186+0000', 'mapping_epoch': 480, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_clean_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-12T15:35:19.331209+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-10-11T11:19:35.850 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-10-11T11:19:35.850 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 516, 'reported_epoch': 499, 'state': 'active+undersized', 'last_fresh': '2023-10-11T11:19:00.636960+0000', 'last_change': '2023-10-11T11:15:31.679465+0000', 'last_active': '2023-10-11T11:19:00.636960+0000', 'last_peered': '2023-10-11T11:19:00.636960+0000', 'last_clean': '2023-10-11T11:15:27.638749+0000', 'last_became_active': '2023-10-11T11:15:31.679465+0000', 'last_became_peered': '2023-10-11T11:15:31.679465+0000', 'last_unstale': '2023-10-11T11:19:00.636960+0000', 'last_undegraded': '2023-10-11T11:19:00.636960+0000', 'last_fullsized': '2023-10-11T11:15:31.659135+0000', 'mapping_epoch': 480, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_clean_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-12T16:03:42.656692+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-10-11T11:19:35.851 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-10-11T11:19:36.247 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T11:19:36.247 INFO:teuthology.orchestra.run.smithi111.stdout:{"fsid":"022ce6f2-9a5b-45ad-8b77-a1332d92f228","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2171,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":500,"num_osds":3,"num_up_osds":2,"osd_up_since":1697022930,"num_in_osds":3,"osd_in_since":1697021005,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":2},{"state_name":"active+undersized+degraded","count":1},{"state_name":"active+clean+remapped","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3170594816,"bytes_avail":286739697664,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":36,"modified":"2023-10-11T11:19:04.644408+0000","services":{}},"progress_events":{}} 2023-10-11T11:19:36.261 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-10-11T11:19:36.262 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-10-11T11:19:36.576 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T11:19:36.579 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2023-10-11T11:19:36.590 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":1578,"stamp":"2023-10-11T11:19:34.651828+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":11,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":107,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3096284,"kb_used_data":418996,"kb_used_omap":292,"kb_used_meta":2676955,"kb_avail":280019236,"statfs":{"total":289910292480,"available":286739697664,"internally_reserved":0,"allocated":429051904,"data_stored":420453125,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299441,"internal_metadata":2741202511},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002857"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":517,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-10-11T11:19:02.772828+0000","last_change":"2023-10-11T11:15:31.661747+0000","last_active":"2023-10-11T11:19:02.772828+0000","last_peered":"2023-10-11T11:19:02.772828+0000","last_clean":"2023-10-11T11:15:27.638215+0000","last_became_active":"2023-10-11T11:15:31.661747+0000","last_became_peered":"2023-10-11T11:15:31.661747+0000","last_unstale":"2023-10-11T11:19:02.772828+0000","last_undegraded":"2023-10-11T11:19:02.772828+0000","last_fullsized":"2023-10-11T11:15:31.649336+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:15:25.047039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":28,"reported_epoch":499,"state":"active+undersized","last_fresh":"2023-10-11T11:19:00.628569+0000","last_change":"2023-10-11T11:15:31.671363+0000","last_active":"2023-10-11T11:19:00.628569+0000","last_peered":"2023-10-11T11:19:00.628569+0000","last_clean":"2023-10-11T10:43:40.751300+0000","last_became_active":"2023-10-11T11:15:31.671363+0000","last_became_peered":"2023-10-11T11:15:31.671363+0000","last_unstale":"2023-10-11T11:19:00.628569+0000","last_undegraded":"2023-10-11T11:19:00.628569+0000","last_fullsized":"2023-10-11T11:15:31.652001+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:01:53.884964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":512,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-10-11T11:19:03.429640+0000","last_change":"2023-10-11T10:43:41.772781+0000","last_active":"2023-10-11T11:19:03.429640+0000","last_peered":"2023-10-11T11:19:03.429640+0000","last_clean":"2023-10-11T11:19:03.429640+0000","last_became_active":"2023-10-11T10:43:41.772443+0000","last_became_peered":"2023-10-11T10:43:41.772443+0000","last_unstale":"2023-10-11T11:19:03.429640+0000","last_undegraded":"2023-10-11T11:19:03.429640+0000","last_fullsized":"2023-10-11T11:19:03.429640+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T14:15:30.547284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":513,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-10-11T11:19:02.772211+0000","last_change":"2023-10-11T10:43:41.772615+0000","last_active":"2023-10-11T11:19:02.772211+0000","last_peered":"2023-10-11T11:19:02.772211+0000","last_clean":"2023-10-11T11:19:02.772211+0000","last_became_active":"2023-10-11T10:43:41.772290+0000","last_became_peered":"2023-10-11T10:43:41.772290+0000","last_unstale":"2023-10-11T11:19:02.772211+0000","last_undegraded":"2023-10-11T11:19:02.772211+0000","last_fullsized":"2023-10-11T11:19:02.772211+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:26:06.909963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":647,"reported_epoch":499,"state":"active+undersized+degraded","last_fresh":"2023-10-11T11:19:00.625113+0000","last_change":"2023-10-11T11:15:31.654022+0000","last_active":"2023-10-11T11:19:00.625113+0000","last_peered":"2023-10-11T11:19:00.625113+0000","last_clean":"2023-10-11T11:15:27.636884+0000","last_became_active":"2023-10-11T11:15:31.654022+0000","last_became_peered":"2023-10-11T11:15:31.654022+0000","last_unstale":"2023-10-11T11:19:00.625113+0000","last_undegraded":"2023-10-11T11:15:31.646946+0000","last_fullsized":"2023-10-11T11:15:31.646751+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:36.739637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:36.739637+0000","last_clean_scrub_stamp":"2023-10-11T10:43:36.739637+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T11:33:33.696447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":519,"reported_epoch":499,"state":"active+clean+remapped","last_fresh":"2023-10-11T11:19:01.882635+0000","last_change":"2023-10-11T11:15:33.660200+0000","last_active":"2023-10-11T11:19:01.882635+0000","last_peered":"2023-10-11T11:19:01.882635+0000","last_clean":"2023-10-11T11:19:01.882635+0000","last_became_active":"2023-10-11T11:15:33.659619+0000","last_became_peered":"2023-10-11T11:15:33.659619+0000","last_unstale":"2023-10-11T11:19:01.882635+0000","last_undegraded":"2023-10-11T11:19:01.882635+0000","last_fullsized":"2023-10-11T11:19:01.882635+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T17:21:16.866392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":517,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-10-11T11:19:02.775128+0000","last_change":"2023-10-11T11:15:31.670813+0000","last_active":"2023-10-11T11:19:02.775128+0000","last_peered":"2023-10-11T11:19:02.775128+0000","last_clean":"2023-10-11T11:15:27.638219+0000","last_became_active":"2023-10-11T11:15:31.670813+0000","last_became_peered":"2023-10-11T11:15:31.670813+0000","last_unstale":"2023-10-11T11:19:02.775128+0000","last_undegraded":"2023-10-11T11:19:02.775128+0000","last_fullsized":"2023-10-11T11:15:31.658785+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:41:50.872346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":516,"reported_epoch":499,"state":"active+undersized","last_fresh":"2023-10-11T11:19:00.631668+0000","last_change":"2023-10-11T11:15:31.666378+0000","last_active":"2023-10-11T11:19:00.631668+0000","last_peered":"2023-10-11T11:19:00.631668+0000","last_clean":"2023-10-11T11:15:27.638303+0000","last_became_active":"2023-10-11T11:15:31.666378+0000","last_became_peered":"2023-10-11T11:15:31.666378+0000","last_unstale":"2023-10-11T11:19:00.631668+0000","last_undegraded":"2023-10-11T11:19:00.631668+0000","last_fullsized":"2023-10-11T11:15:31.649186+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T15:35:19.331209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":516,"reported_epoch":499,"state":"active+undersized","last_fresh":"2023-10-11T11:19:00.636960+0000","last_change":"2023-10-11T11:15:31.679465+0000","last_active":"2023-10-11T11:19:00.636960+0000","last_peered":"2023-10-11T11:19:00.636960+0000","last_clean":"2023-10-11T11:15:27.638749+0000","last_became_active":"2023-10-11T11:15:31.679465+0000","last_became_peered":"2023-10-11T11:15:31.679465+0000","last_unstale":"2023-10-11T11:19:00.636960+0000","last_undegraded":"2023-10-11T11:19:00.636960+0000","last_fullsized":"2023-10-11T11:15:31.659135+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T16:03:42.656692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":11,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738802,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":785348,"kb_used_data":5812,"kb_used_omap":96,"kb_used_meta":779423,"kb_avail":93586492,"statfs":{"total":96636764160,"available":95832567808,"internally_reserved":0,"allocated":5951488,"data_stored":3106918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":98609,"internal_metadata":798129871},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 11:14:37 2023","interfaces":[{"interface":"back","average":{"1min":0.314,"5min":0.316,"15min":0.443},"min":{"1min":0.24399999999999999,"5min":0.14799999999999999,"15min":0.14599999999999999},"max":{"1min":0.48499999999999999,"5min":1.712,"15min":9.6400000000000006},"last":0.31},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.35499999999999998,"15min":0.41299999999999998},"min":{"1min":0.20899999999999999,"5min":0.17199999999999999,"15min":0.16400000000000001},"max":{"1min":3.8180000000000001,"5min":3.8180000000000001,"15min":6.3200000000000003},"last":0.26500000000000001}]},{"osd":1,"last update":"Wed Oct 11 11:18:39 2023","interfaces":[{"interface":"back","average":{"1min":0.27500000000000002,"5min":0.29299999999999998,"15min":0.32400000000000001},"min":{"1min":0.154,"5min":0.154,"15min":0.154},"max":{"1min":0.41999999999999998,"5min":0.59299999999999997,"15min":2.9540000000000002},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.307,"5min":0.35899999999999999,"15min":0.36499999999999999},"min":{"1min":0.18099999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.45100000000000001,"5min":3.2719999999999998,"15min":3.2719999999999998},"last":0.45000000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738756,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1484496,"kb_used_data":406792,"kb_used_omap":107,"kb_used_meta":1077588,"kb_avail":92887344,"statfs":{"total":96636764160,"available":95116640256,"internally_reserved":0,"allocated":416555008,"data_stored":413648921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110309,"internal_metadata":1103450395},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Oct 11 11:14:30 2023","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.35399999999999998,"15min":0.40600000000000003},"min":{"1min":0.18099999999999999,"5min":0.155,"15min":0.14899999999999999},"max":{"1min":1.4319999999999999,"5min":2.3460000000000001,"15min":6.593},"last":0.26200000000000001},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.39100000000000001,"15min":0.48199999999999998},"min":{"1min":0.189,"5min":0.189,"15min":0.183},"max":{"1min":2.2120000000000002,"5min":2.4089999999999998,"15min":6.5960000000000001},"last":0.316}]},{"osd":2,"last update":"Wed Oct 11 11:14:30 2023","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.35499999999999998,"15min":0.47099999999999997},"min":{"1min":0.20300000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":1.2490000000000001,"5min":2.665,"15min":8.4730000000000008},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.375,"15min":0.442},"min":{"1min":0.23200000000000001,"5min":0.20499999999999999,"15min":0.19600000000000001},"max":{"1min":1.28,"5min":2.6230000000000002,"15min":7.0129999999999999},"last":0.33600000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738803,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":826440,"kb_used_data":6392,"kb_used_omap":88,"kb_used_meta":819943,"kb_avail":93545400,"statfs":{"total":96636764160,"available":95790489600,"internally_reserved":0,"allocated":6545408,"data_stored":3697286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":90523,"internal_metadata":839622245},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 11:14:33 2023","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.42799999999999999,"15min":0.41099999999999998},"min":{"1min":0.151,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":1.2809999999999999,"5min":3.5739999999999998,"15min":8.2479999999999993},"last":0.223},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.48299999999999998,"15min":0.49199999999999999},"min":{"1min":0.20699999999999999,"5min":0.182,"15min":0.17999999999999999},"max":{"1min":1.323,"5min":3.6080000000000001,"15min":8.2870000000000008},"last":0.28399999999999997}]},{"osd":2,"last update":"Wed Oct 11 11:18:40 2023","interfaces":[{"interface":"back","average":{"1min":0.33400000000000002,"5min":0.378,"15min":0.35599999999999998},"min":{"1min":0.16500000000000001,"5min":0.14699999999999999,"15min":0.14699999999999999},"max":{"1min":1.244,"5min":7.4829999999999997,"15min":10.617000000000001},"last":0.371},{"interface":"front","average":{"1min":0.34599999999999997,"5min":0.39000000000000001,"15min":0.34999999999999998},"min":{"1min":0.20399999999999999,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.81100000000000005,"5min":7.5350000000000001,"15min":7.5350000000000001},"last":0.41999999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-11T11:19:39.592 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-10-11T11:19:39.922 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T11:19:39.924 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2023-10-11T11:19:39.935 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":1580,"stamp":"2023-10-11T11:19:38.652886+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":11,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":107,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3096284,"kb_used_data":418996,"kb_used_omap":292,"kb_used_meta":2676955,"kb_avail":280019236,"statfs":{"total":289910292480,"available":286739697664,"internally_reserved":0,"allocated":429051904,"data_stored":420453125,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299441,"internal_metadata":2741202511},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002921"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":517,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-10-11T11:19:02.772828+0000","last_change":"2023-10-11T11:15:31.661747+0000","last_active":"2023-10-11T11:19:02.772828+0000","last_peered":"2023-10-11T11:19:02.772828+0000","last_clean":"2023-10-11T11:15:27.638215+0000","last_became_active":"2023-10-11T11:15:31.661747+0000","last_became_peered":"2023-10-11T11:15:31.661747+0000","last_unstale":"2023-10-11T11:19:02.772828+0000","last_undegraded":"2023-10-11T11:19:02.772828+0000","last_fullsized":"2023-10-11T11:15:31.649336+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:15:25.047039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":28,"reported_epoch":499,"state":"active+undersized","last_fresh":"2023-10-11T11:19:00.628569+0000","last_change":"2023-10-11T11:15:31.671363+0000","last_active":"2023-10-11T11:19:00.628569+0000","last_peered":"2023-10-11T11:19:00.628569+0000","last_clean":"2023-10-11T10:43:40.751300+0000","last_became_active":"2023-10-11T11:15:31.671363+0000","last_became_peered":"2023-10-11T11:15:31.671363+0000","last_unstale":"2023-10-11T11:19:00.628569+0000","last_undegraded":"2023-10-11T11:19:00.628569+0000","last_fullsized":"2023-10-11T11:15:31.652001+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:01:53.884964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":512,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-10-11T11:19:03.429640+0000","last_change":"2023-10-11T10:43:41.772781+0000","last_active":"2023-10-11T11:19:03.429640+0000","last_peered":"2023-10-11T11:19:03.429640+0000","last_clean":"2023-10-11T11:19:03.429640+0000","last_became_active":"2023-10-11T10:43:41.772443+0000","last_became_peered":"2023-10-11T10:43:41.772443+0000","last_unstale":"2023-10-11T11:19:03.429640+0000","last_undegraded":"2023-10-11T11:19:03.429640+0000","last_fullsized":"2023-10-11T11:19:03.429640+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T14:15:30.547284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":513,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-10-11T11:19:02.772211+0000","last_change":"2023-10-11T10:43:41.772615+0000","last_active":"2023-10-11T11:19:02.772211+0000","last_peered":"2023-10-11T11:19:02.772211+0000","last_clean":"2023-10-11T11:19:02.772211+0000","last_became_active":"2023-10-11T10:43:41.772290+0000","last_became_peered":"2023-10-11T10:43:41.772290+0000","last_unstale":"2023-10-11T11:19:02.772211+0000","last_undegraded":"2023-10-11T11:19:02.772211+0000","last_fullsized":"2023-10-11T11:19:02.772211+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:26:06.909963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":647,"reported_epoch":499,"state":"active+undersized+degraded","last_fresh":"2023-10-11T11:19:00.625113+0000","last_change":"2023-10-11T11:15:31.654022+0000","last_active":"2023-10-11T11:19:00.625113+0000","last_peered":"2023-10-11T11:19:00.625113+0000","last_clean":"2023-10-11T11:15:27.636884+0000","last_became_active":"2023-10-11T11:15:31.654022+0000","last_became_peered":"2023-10-11T11:15:31.654022+0000","last_unstale":"2023-10-11T11:19:00.625113+0000","last_undegraded":"2023-10-11T11:15:31.646946+0000","last_fullsized":"2023-10-11T11:15:31.646751+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:36.739637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:36.739637+0000","last_clean_scrub_stamp":"2023-10-11T10:43:36.739637+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T11:33:33.696447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":519,"reported_epoch":499,"state":"active+clean+remapped","last_fresh":"2023-10-11T11:19:01.882635+0000","last_change":"2023-10-11T11:15:33.660200+0000","last_active":"2023-10-11T11:19:01.882635+0000","last_peered":"2023-10-11T11:19:01.882635+0000","last_clean":"2023-10-11T11:19:01.882635+0000","last_became_active":"2023-10-11T11:15:33.659619+0000","last_became_peered":"2023-10-11T11:15:33.659619+0000","last_unstale":"2023-10-11T11:19:01.882635+0000","last_undegraded":"2023-10-11T11:19:01.882635+0000","last_fullsized":"2023-10-11T11:19:01.882635+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T17:21:16.866392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":517,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-10-11T11:19:02.775128+0000","last_change":"2023-10-11T11:15:31.670813+0000","last_active":"2023-10-11T11:19:02.775128+0000","last_peered":"2023-10-11T11:19:02.775128+0000","last_clean":"2023-10-11T11:15:27.638219+0000","last_became_active":"2023-10-11T11:15:31.670813+0000","last_became_peered":"2023-10-11T11:15:31.670813+0000","last_unstale":"2023-10-11T11:19:02.775128+0000","last_undegraded":"2023-10-11T11:19:02.775128+0000","last_fullsized":"2023-10-11T11:15:31.658785+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:41:50.872346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":516,"reported_epoch":499,"state":"active+undersized","last_fresh":"2023-10-11T11:19:00.631668+0000","last_change":"2023-10-11T11:15:31.666378+0000","last_active":"2023-10-11T11:19:00.631668+0000","last_peered":"2023-10-11T11:19:00.631668+0000","last_clean":"2023-10-11T11:15:27.638303+0000","last_became_active":"2023-10-11T11:15:31.666378+0000","last_became_peered":"2023-10-11T11:15:31.666378+0000","last_unstale":"2023-10-11T11:19:00.631668+0000","last_undegraded":"2023-10-11T11:19:00.631668+0000","last_fullsized":"2023-10-11T11:15:31.649186+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T15:35:19.331209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":516,"reported_epoch":499,"state":"active+undersized","last_fresh":"2023-10-11T11:19:00.636960+0000","last_change":"2023-10-11T11:15:31.679465+0000","last_active":"2023-10-11T11:19:00.636960+0000","last_peered":"2023-10-11T11:19:00.636960+0000","last_clean":"2023-10-11T11:15:27.638749+0000","last_became_active":"2023-10-11T11:15:31.679465+0000","last_became_peered":"2023-10-11T11:15:31.679465+0000","last_unstale":"2023-10-11T11:19:00.636960+0000","last_undegraded":"2023-10-11T11:19:00.636960+0000","last_fullsized":"2023-10-11T11:15:31.659135+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T16:03:42.656692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":11,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738803,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":785348,"kb_used_data":5812,"kb_used_omap":96,"kb_used_meta":779423,"kb_avail":93586492,"statfs":{"total":96636764160,"available":95832567808,"internally_reserved":0,"allocated":5951488,"data_stored":3106918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":98609,"internal_metadata":798129871},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 11:14:37 2023","interfaces":[{"interface":"back","average":{"1min":0.314,"5min":0.316,"15min":0.443},"min":{"1min":0.24399999999999999,"5min":0.14799999999999999,"15min":0.14599999999999999},"max":{"1min":0.48499999999999999,"5min":1.712,"15min":9.6400000000000006},"last":0.31},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.35499999999999998,"15min":0.41299999999999998},"min":{"1min":0.20899999999999999,"5min":0.17199999999999999,"15min":0.16400000000000001},"max":{"1min":3.8180000000000001,"5min":3.8180000000000001,"15min":6.3200000000000003},"last":0.26500000000000001}]},{"osd":1,"last update":"Wed Oct 11 11:18:39 2023","interfaces":[{"interface":"back","average":{"1min":0.27500000000000002,"5min":0.29299999999999998,"15min":0.32400000000000001},"min":{"1min":0.154,"5min":0.154,"15min":0.154},"max":{"1min":0.41999999999999998,"5min":0.59299999999999997,"15min":2.9540000000000002},"last":0.33400000000000002},{"interface":"front","average":{"1min":0.307,"5min":0.35899999999999999,"15min":0.36499999999999999},"min":{"1min":0.18099999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.45100000000000001,"5min":3.2719999999999998,"15min":3.2719999999999998},"last":0.28899999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738756,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1484496,"kb_used_data":406792,"kb_used_omap":107,"kb_used_meta":1077588,"kb_avail":92887344,"statfs":{"total":96636764160,"available":95116640256,"internally_reserved":0,"allocated":416555008,"data_stored":413648921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110309,"internal_metadata":1103450395},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Oct 11 11:14:30 2023","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.35399999999999998,"15min":0.40600000000000003},"min":{"1min":0.18099999999999999,"5min":0.155,"15min":0.14899999999999999},"max":{"1min":1.4319999999999999,"5min":2.3460000000000001,"15min":6.593},"last":0.26200000000000001},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.39100000000000001,"15min":0.48199999999999998},"min":{"1min":0.189,"5min":0.189,"15min":0.183},"max":{"1min":2.2120000000000002,"5min":2.4089999999999998,"15min":6.5960000000000001},"last":0.316}]},{"osd":2,"last update":"Wed Oct 11 11:14:30 2023","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.35499999999999998,"15min":0.47099999999999997},"min":{"1min":0.20300000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":1.2490000000000001,"5min":2.665,"15min":8.4730000000000008},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.375,"15min":0.442},"min":{"1min":0.23200000000000001,"5min":0.20499999999999999,"15min":0.19600000000000001},"max":{"1min":1.28,"5min":2.6230000000000002,"15min":7.0129999999999999},"last":0.33600000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738804,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":826440,"kb_used_data":6392,"kb_used_omap":88,"kb_used_meta":819943,"kb_avail":93545400,"statfs":{"total":96636764160,"available":95790489600,"internally_reserved":0,"allocated":6545408,"data_stored":3697286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":90523,"internal_metadata":839622245},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 11:14:33 2023","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.42799999999999999,"15min":0.41099999999999998},"min":{"1min":0.151,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":1.2809999999999999,"5min":3.5739999999999998,"15min":8.2479999999999993},"last":0.223},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.48299999999999998,"15min":0.49199999999999999},"min":{"1min":0.20699999999999999,"5min":0.182,"15min":0.17999999999999999},"max":{"1min":1.323,"5min":3.6080000000000001,"15min":8.2870000000000008},"last":0.28399999999999997}]},{"osd":2,"last update":"Wed Oct 11 11:18:40 2023","interfaces":[{"interface":"back","average":{"1min":0.33400000000000002,"5min":0.378,"15min":0.35599999999999998},"min":{"1min":0.16500000000000001,"5min":0.14699999999999999,"15min":0.14699999999999999},"max":{"1min":1.244,"5min":7.4829999999999997,"15min":10.617000000000001},"last":0.24199999999999999},{"interface":"front","average":{"1min":0.34599999999999997,"5min":0.39000000000000001,"15min":0.34999999999999998},"min":{"1min":0.20399999999999999,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.81100000000000005,"5min":7.5350000000000001,"15min":7.5350000000000001},"last":0.215}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-11T11:19:39.937 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-10-11T11:19:40.250 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T11:19:40.252 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2023-10-11T11:19:40.264 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":1580,"stamp":"2023-10-11T11:19:38.652886+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":11,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":107,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3096284,"kb_used_data":418996,"kb_used_omap":292,"kb_used_meta":2676955,"kb_avail":280019236,"statfs":{"total":289910292480,"available":286739697664,"internally_reserved":0,"allocated":429051904,"data_stored":420453125,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299441,"internal_metadata":2741202511},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002921"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":517,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-10-11T11:19:02.772828+0000","last_change":"2023-10-11T11:15:31.661747+0000","last_active":"2023-10-11T11:19:02.772828+0000","last_peered":"2023-10-11T11:19:02.772828+0000","last_clean":"2023-10-11T11:15:27.638215+0000","last_became_active":"2023-10-11T11:15:31.661747+0000","last_became_peered":"2023-10-11T11:15:31.661747+0000","last_unstale":"2023-10-11T11:19:02.772828+0000","last_undegraded":"2023-10-11T11:19:02.772828+0000","last_fullsized":"2023-10-11T11:15:31.649336+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:15:25.047039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":28,"reported_epoch":499,"state":"active+undersized","last_fresh":"2023-10-11T11:19:00.628569+0000","last_change":"2023-10-11T11:15:31.671363+0000","last_active":"2023-10-11T11:19:00.628569+0000","last_peered":"2023-10-11T11:19:00.628569+0000","last_clean":"2023-10-11T10:43:40.751300+0000","last_became_active":"2023-10-11T11:15:31.671363+0000","last_became_peered":"2023-10-11T11:15:31.671363+0000","last_unstale":"2023-10-11T11:19:00.628569+0000","last_undegraded":"2023-10-11T11:19:00.628569+0000","last_fullsized":"2023-10-11T11:15:31.652001+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:01:53.884964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":512,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-10-11T11:19:03.429640+0000","last_change":"2023-10-11T10:43:41.772781+0000","last_active":"2023-10-11T11:19:03.429640+0000","last_peered":"2023-10-11T11:19:03.429640+0000","last_clean":"2023-10-11T11:19:03.429640+0000","last_became_active":"2023-10-11T10:43:41.772443+0000","last_became_peered":"2023-10-11T10:43:41.772443+0000","last_unstale":"2023-10-11T11:19:03.429640+0000","last_undegraded":"2023-10-11T11:19:03.429640+0000","last_fullsized":"2023-10-11T11:19:03.429640+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T14:15:30.547284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":513,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-10-11T11:19:02.772211+0000","last_change":"2023-10-11T10:43:41.772615+0000","last_active":"2023-10-11T11:19:02.772211+0000","last_peered":"2023-10-11T11:19:02.772211+0000","last_clean":"2023-10-11T11:19:02.772211+0000","last_became_active":"2023-10-11T10:43:41.772290+0000","last_became_peered":"2023-10-11T10:43:41.772290+0000","last_unstale":"2023-10-11T11:19:02.772211+0000","last_undegraded":"2023-10-11T11:19:02.772211+0000","last_fullsized":"2023-10-11T11:19:02.772211+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:26:06.909963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":647,"reported_epoch":499,"state":"active+undersized+degraded","last_fresh":"2023-10-11T11:19:00.625113+0000","last_change":"2023-10-11T11:15:31.654022+0000","last_active":"2023-10-11T11:19:00.625113+0000","last_peered":"2023-10-11T11:19:00.625113+0000","last_clean":"2023-10-11T11:15:27.636884+0000","last_became_active":"2023-10-11T11:15:31.654022+0000","last_became_peered":"2023-10-11T11:15:31.654022+0000","last_unstale":"2023-10-11T11:19:00.625113+0000","last_undegraded":"2023-10-11T11:15:31.646946+0000","last_fullsized":"2023-10-11T11:15:31.646751+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:36.739637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:36.739637+0000","last_clean_scrub_stamp":"2023-10-11T10:43:36.739637+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T11:33:33.696447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":519,"reported_epoch":499,"state":"active+clean+remapped","last_fresh":"2023-10-11T11:19:01.882635+0000","last_change":"2023-10-11T11:15:33.660200+0000","last_active":"2023-10-11T11:19:01.882635+0000","last_peered":"2023-10-11T11:19:01.882635+0000","last_clean":"2023-10-11T11:19:01.882635+0000","last_became_active":"2023-10-11T11:15:33.659619+0000","last_became_peered":"2023-10-11T11:15:33.659619+0000","last_unstale":"2023-10-11T11:19:01.882635+0000","last_undegraded":"2023-10-11T11:19:01.882635+0000","last_fullsized":"2023-10-11T11:19:01.882635+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T17:21:16.866392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":517,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-10-11T11:19:02.775128+0000","last_change":"2023-10-11T11:15:31.670813+0000","last_active":"2023-10-11T11:19:02.775128+0000","last_peered":"2023-10-11T11:19:02.775128+0000","last_clean":"2023-10-11T11:15:27.638219+0000","last_became_active":"2023-10-11T11:15:31.670813+0000","last_became_peered":"2023-10-11T11:15:31.670813+0000","last_unstale":"2023-10-11T11:19:02.775128+0000","last_undegraded":"2023-10-11T11:19:02.775128+0000","last_fullsized":"2023-10-11T11:15:31.658785+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:41:50.872346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":516,"reported_epoch":499,"state":"active+undersized","last_fresh":"2023-10-11T11:19:00.631668+0000","last_change":"2023-10-11T11:15:31.666378+0000","last_active":"2023-10-11T11:19:00.631668+0000","last_peered":"2023-10-11T11:19:00.631668+0000","last_clean":"2023-10-11T11:15:27.638303+0000","last_became_active":"2023-10-11T11:15:31.666378+0000","last_became_peered":"2023-10-11T11:15:31.666378+0000","last_unstale":"2023-10-11T11:19:00.631668+0000","last_undegraded":"2023-10-11T11:19:00.631668+0000","last_fullsized":"2023-10-11T11:15:31.649186+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T15:35:19.331209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":516,"reported_epoch":499,"state":"active+undersized","last_fresh":"2023-10-11T11:19:00.636960+0000","last_change":"2023-10-11T11:15:31.679465+0000","last_active":"2023-10-11T11:19:00.636960+0000","last_peered":"2023-10-11T11:19:00.636960+0000","last_clean":"2023-10-11T11:15:27.638749+0000","last_became_active":"2023-10-11T11:15:31.679465+0000","last_became_peered":"2023-10-11T11:15:31.679465+0000","last_unstale":"2023-10-11T11:19:00.636960+0000","last_undegraded":"2023-10-11T11:19:00.636960+0000","last_fullsized":"2023-10-11T11:15:31.659135+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T16:03:42.656692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":11,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738803,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":785348,"kb_used_data":5812,"kb_used_omap":96,"kb_used_meta":779423,"kb_avail":93586492,"statfs":{"total":96636764160,"available":95832567808,"internally_reserved":0,"allocated":5951488,"data_stored":3106918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":98609,"internal_metadata":798129871},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 11:14:37 2023","interfaces":[{"interface":"back","average":{"1min":0.314,"5min":0.316,"15min":0.443},"min":{"1min":0.24399999999999999,"5min":0.14799999999999999,"15min":0.14599999999999999},"max":{"1min":0.48499999999999999,"5min":1.712,"15min":9.6400000000000006},"last":0.31},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.35499999999999998,"15min":0.41299999999999998},"min":{"1min":0.20899999999999999,"5min":0.17199999999999999,"15min":0.16400000000000001},"max":{"1min":3.8180000000000001,"5min":3.8180000000000001,"15min":6.3200000000000003},"last":0.26500000000000001}]},{"osd":1,"last update":"Wed Oct 11 11:18:39 2023","interfaces":[{"interface":"back","average":{"1min":0.27500000000000002,"5min":0.29299999999999998,"15min":0.32400000000000001},"min":{"1min":0.154,"5min":0.154,"15min":0.154},"max":{"1min":0.41999999999999998,"5min":0.59299999999999997,"15min":2.9540000000000002},"last":0.33400000000000002},{"interface":"front","average":{"1min":0.307,"5min":0.35899999999999999,"15min":0.36499999999999999},"min":{"1min":0.18099999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.45100000000000001,"5min":3.2719999999999998,"15min":3.2719999999999998},"last":0.28899999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738756,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1484496,"kb_used_data":406792,"kb_used_omap":107,"kb_used_meta":1077588,"kb_avail":92887344,"statfs":{"total":96636764160,"available":95116640256,"internally_reserved":0,"allocated":416555008,"data_stored":413648921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110309,"internal_metadata":1103450395},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Oct 11 11:14:30 2023","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.35399999999999998,"15min":0.40600000000000003},"min":{"1min":0.18099999999999999,"5min":0.155,"15min":0.14899999999999999},"max":{"1min":1.4319999999999999,"5min":2.3460000000000001,"15min":6.593},"last":0.26200000000000001},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.39100000000000001,"15min":0.48199999999999998},"min":{"1min":0.189,"5min":0.189,"15min":0.183},"max":{"1min":2.2120000000000002,"5min":2.4089999999999998,"15min":6.5960000000000001},"last":0.316}]},{"osd":2,"last update":"Wed Oct 11 11:14:30 2023","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.35499999999999998,"15min":0.47099999999999997},"min":{"1min":0.20300000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":1.2490000000000001,"5min":2.665,"15min":8.4730000000000008},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.375,"15min":0.442},"min":{"1min":0.23200000000000001,"5min":0.20499999999999999,"15min":0.19600000000000001},"max":{"1min":1.28,"5min":2.6230000000000002,"15min":7.0129999999999999},"last":0.33600000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738804,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":826440,"kb_used_data":6392,"kb_used_omap":88,"kb_used_meta":819943,"kb_avail":93545400,"statfs":{"total":96636764160,"available":95790489600,"internally_reserved":0,"allocated":6545408,"data_stored":3697286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":90523,"internal_metadata":839622245},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 11:14:33 2023","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.42799999999999999,"15min":0.41099999999999998},"min":{"1min":0.151,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":1.2809999999999999,"5min":3.5739999999999998,"15min":8.2479999999999993},"last":0.223},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.48299999999999998,"15min":0.49199999999999999},"min":{"1min":0.20699999999999999,"5min":0.182,"15min":0.17999999999999999},"max":{"1min":1.323,"5min":3.6080000000000001,"15min":8.2870000000000008},"last":0.28399999999999997}]},{"osd":2,"last update":"Wed Oct 11 11:18:40 2023","interfaces":[{"interface":"back","average":{"1min":0.33400000000000002,"5min":0.378,"15min":0.35599999999999998},"min":{"1min":0.16500000000000001,"5min":0.14699999999999999,"15min":0.14699999999999999},"max":{"1min":1.244,"5min":7.4829999999999997,"15min":10.617000000000001},"last":0.24199999999999999},{"interface":"front","average":{"1min":0.34599999999999997,"5min":0.39000000000000001,"15min":0.34999999999999998},"min":{"1min":0.20399999999999999,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.81100000000000005,"5min":7.5350000000000001,"15min":7.5350000000000001},"last":0.215}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-11T11:19:40.265 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-10-11T11:19:40.265 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 517, 'reported_epoch': 500, 'state': 'active+undersized', 'last_fresh': '2023-10-11T11:19:02.772828+0000', 'last_change': '2023-10-11T11:15:31.661747+0000', 'last_active': '2023-10-11T11:19:02.772828+0000', 'last_peered': '2023-10-11T11:19:02.772828+0000', 'last_clean': '2023-10-11T11:15:27.638215+0000', 'last_became_active': '2023-10-11T11:15:31.661747+0000', 'last_became_peered': '2023-10-11T11:15:31.661747+0000', 'last_unstale': '2023-10-11T11:19:02.772828+0000', 'last_undegraded': '2023-10-11T11:19:02.772828+0000', 'last_fullsized': '2023-10-11T11:15:31.649336+0000', 'mapping_epoch': 480, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_clean_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-12T22:15:25.047039+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-10-11T11:19:40.266 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-10-11T11:19:40.266 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 28, 'reported_epoch': 499, 'state': 'active+undersized', 'last_fresh': '2023-10-11T11:19:00.628569+0000', 'last_change': '2023-10-11T11:15:31.671363+0000', 'last_active': '2023-10-11T11:19:00.628569+0000', 'last_peered': '2023-10-11T11:19:00.628569+0000', 'last_clean': '2023-10-11T10:43:40.751300+0000', 'last_became_active': '2023-10-11T11:15:31.671363+0000', 'last_became_peered': '2023-10-11T11:15:31.671363+0000', 'last_unstale': '2023-10-11T11:19:00.628569+0000', 'last_undegraded': '2023-10-11T11:19:00.628569+0000', 'last_fullsized': '2023-10-11T11:15:31.652001+0000', 'mapping_epoch': 480, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_clean_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-12T21:01:53.884964+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-10-11T11:19:40.266 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-10-11T11:19:40.267 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "11'97", 'reported_seq': 647, 'reported_epoch': 499, 'state': 'active+undersized+degraded', 'last_fresh': '2023-10-11T11:19:00.625113+0000', 'last_change': '2023-10-11T11:15:31.654022+0000', 'last_active': '2023-10-11T11:19:00.625113+0000', 'last_peered': '2023-10-11T11:19:00.625113+0000', 'last_clean': '2023-10-11T11:15:27.636884+0000', 'last_became_active': '2023-10-11T11:15:31.654022+0000', 'last_became_peered': '2023-10-11T11:15:31.654022+0000', 'last_unstale': '2023-10-11T11:19:00.625113+0000', 'last_undegraded': '2023-10-11T11:15:31.646946+0000', 'last_fullsized': '2023-10-11T11:15:31.646751+0000', 'mapping_epoch': 480, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-11T10:43:36.739637+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-11T10:43:36.739637+0000', 'last_clean_scrub_stamp': '2023-10-11T10:43:36.739637+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-12T11:33:33.696447+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-10-11T11:19:40.267 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.0 is not active+clean 2023-10-11T11:19:40.267 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 519, 'reported_epoch': 499, 'state': 'active+clean+remapped', 'last_fresh': '2023-10-11T11:19:01.882635+0000', 'last_change': '2023-10-11T11:15:33.660200+0000', 'last_active': '2023-10-11T11:19:01.882635+0000', 'last_peered': '2023-10-11T11:19:01.882635+0000', 'last_clean': '2023-10-11T11:19:01.882635+0000', 'last_became_active': '2023-10-11T11:15:33.659619+0000', 'last_became_peered': '2023-10-11T11:15:33.659619+0000', 'last_unstale': '2023-10-11T11:19:01.882635+0000', 'last_undegraded': '2023-10-11T11:19:01.882635+0000', 'last_fullsized': '2023-10-11T11:19:01.882635+0000', 'mapping_epoch': 482, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 483, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_clean_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-12T17:21:16.866392+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2, 1], 'avail_no_missing': ['2', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-10-11T11:19:40.268 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-10-11T11:19:40.268 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 517, 'reported_epoch': 500, 'state': 'active+undersized', 'last_fresh': '2023-10-11T11:19:02.775128+0000', 'last_change': '2023-10-11T11:15:31.670813+0000', 'last_active': '2023-10-11T11:19:02.775128+0000', 'last_peered': '2023-10-11T11:19:02.775128+0000', 'last_clean': '2023-10-11T11:15:27.638219+0000', 'last_became_active': '2023-10-11T11:15:31.670813+0000', 'last_became_peered': '2023-10-11T11:15:31.670813+0000', 'last_unstale': '2023-10-11T11:19:02.775128+0000', 'last_undegraded': '2023-10-11T11:19:02.775128+0000', 'last_fullsized': '2023-10-11T11:15:31.658785+0000', 'mapping_epoch': 480, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_clean_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-12T21:41:50.872346+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-10-11T11:19:40.268 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-10-11T11:19:40.268 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 516, 'reported_epoch': 499, 'state': 'active+undersized', 'last_fresh': '2023-10-11T11:19:00.631668+0000', 'last_change': '2023-10-11T11:15:31.666378+0000', 'last_active': '2023-10-11T11:19:00.631668+0000', 'last_peered': '2023-10-11T11:19:00.631668+0000', 'last_clean': '2023-10-11T11:15:27.638303+0000', 'last_became_active': '2023-10-11T11:15:31.666378+0000', 'last_became_peered': '2023-10-11T11:15:31.666378+0000', 'last_unstale': '2023-10-11T11:19:00.631668+0000', 'last_undegraded': '2023-10-11T11:19:00.631668+0000', 'last_fullsized': '2023-10-11T11:15:31.649186+0000', 'mapping_epoch': 480, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_clean_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-12T15:35:19.331209+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-10-11T11:19:40.269 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-10-11T11:19:40.269 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 516, 'reported_epoch': 499, 'state': 'active+undersized', 'last_fresh': '2023-10-11T11:19:00.636960+0000', 'last_change': '2023-10-11T11:15:31.679465+0000', 'last_active': '2023-10-11T11:19:00.636960+0000', 'last_peered': '2023-10-11T11:19:00.636960+0000', 'last_clean': '2023-10-11T11:15:27.638749+0000', 'last_became_active': '2023-10-11T11:15:31.679465+0000', 'last_became_peered': '2023-10-11T11:15:31.679465+0000', 'last_unstale': '2023-10-11T11:19:00.636960+0000', 'last_undegraded': '2023-10-11T11:19:00.636960+0000', 'last_fullsized': '2023-10-11T11:15:31.659135+0000', 'mapping_epoch': 480, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_clean_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-12T16:03:42.656692+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-10-11T11:19:40.269 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-10-11T11:19:40.668 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T11:19:40.668 INFO:teuthology.orchestra.run.smithi111.stdout:{"fsid":"022ce6f2-9a5b-45ad-8b77-a1332d92f228","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2175,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":500,"num_osds":3,"num_up_osds":2,"osd_up_since":1697022930,"num_in_osds":3,"osd_in_since":1697021005,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":2},{"state_name":"active+undersized+degraded","count":1},{"state_name":"active+clean+remapped","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3170594816,"bytes_avail":286739697664,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":36,"modified":"2023-10-11T11:19:04.644408+0000","services":{}},"progress_events":{}} 2023-10-11T11:19:40.682 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-10-11T11:19:40.682 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-10-11T11:19:40.734 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~249s 2023-10-11T11:19:41.000 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T11:19:41.002 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2023-10-11T11:19:41.013 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":1581,"stamp":"2023-10-11T11:19:40.653352+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":11,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":107,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3096284,"kb_used_data":418996,"kb_used_omap":292,"kb_used_meta":2676955,"kb_avail":280019236,"statfs":{"total":289910292480,"available":286739697664,"internally_reserved":0,"allocated":429051904,"data_stored":420453125,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299441,"internal_metadata":2741202511},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002883"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":517,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-10-11T11:19:02.772828+0000","last_change":"2023-10-11T11:15:31.661747+0000","last_active":"2023-10-11T11:19:02.772828+0000","last_peered":"2023-10-11T11:19:02.772828+0000","last_clean":"2023-10-11T11:15:27.638215+0000","last_became_active":"2023-10-11T11:15:31.661747+0000","last_became_peered":"2023-10-11T11:15:31.661747+0000","last_unstale":"2023-10-11T11:19:02.772828+0000","last_undegraded":"2023-10-11T11:19:02.772828+0000","last_fullsized":"2023-10-11T11:15:31.649336+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:15:25.047039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":28,"reported_epoch":499,"state":"active+undersized","last_fresh":"2023-10-11T11:19:00.628569+0000","last_change":"2023-10-11T11:15:31.671363+0000","last_active":"2023-10-11T11:19:00.628569+0000","last_peered":"2023-10-11T11:19:00.628569+0000","last_clean":"2023-10-11T10:43:40.751300+0000","last_became_active":"2023-10-11T11:15:31.671363+0000","last_became_peered":"2023-10-11T11:15:31.671363+0000","last_unstale":"2023-10-11T11:19:00.628569+0000","last_undegraded":"2023-10-11T11:19:00.628569+0000","last_fullsized":"2023-10-11T11:15:31.652001+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:01:53.884964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":512,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-10-11T11:19:03.429640+0000","last_change":"2023-10-11T10:43:41.772781+0000","last_active":"2023-10-11T11:19:03.429640+0000","last_peered":"2023-10-11T11:19:03.429640+0000","last_clean":"2023-10-11T11:19:03.429640+0000","last_became_active":"2023-10-11T10:43:41.772443+0000","last_became_peered":"2023-10-11T10:43:41.772443+0000","last_unstale":"2023-10-11T11:19:03.429640+0000","last_undegraded":"2023-10-11T11:19:03.429640+0000","last_fullsized":"2023-10-11T11:19:03.429640+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T14:15:30.547284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":513,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-10-11T11:19:02.772211+0000","last_change":"2023-10-11T10:43:41.772615+0000","last_active":"2023-10-11T11:19:02.772211+0000","last_peered":"2023-10-11T11:19:02.772211+0000","last_clean":"2023-10-11T11:19:02.772211+0000","last_became_active":"2023-10-11T10:43:41.772290+0000","last_became_peered":"2023-10-11T10:43:41.772290+0000","last_unstale":"2023-10-11T11:19:02.772211+0000","last_undegraded":"2023-10-11T11:19:02.772211+0000","last_fullsized":"2023-10-11T11:19:02.772211+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:26:06.909963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":647,"reported_epoch":499,"state":"active+undersized+degraded","last_fresh":"2023-10-11T11:19:00.625113+0000","last_change":"2023-10-11T11:15:31.654022+0000","last_active":"2023-10-11T11:19:00.625113+0000","last_peered":"2023-10-11T11:19:00.625113+0000","last_clean":"2023-10-11T11:15:27.636884+0000","last_became_active":"2023-10-11T11:15:31.654022+0000","last_became_peered":"2023-10-11T11:15:31.654022+0000","last_unstale":"2023-10-11T11:19:00.625113+0000","last_undegraded":"2023-10-11T11:15:31.646946+0000","last_fullsized":"2023-10-11T11:15:31.646751+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:36.739637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:36.739637+0000","last_clean_scrub_stamp":"2023-10-11T10:43:36.739637+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T11:33:33.696447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":519,"reported_epoch":499,"state":"active+clean+remapped","last_fresh":"2023-10-11T11:19:01.882635+0000","last_change":"2023-10-11T11:15:33.660200+0000","last_active":"2023-10-11T11:19:01.882635+0000","last_peered":"2023-10-11T11:19:01.882635+0000","last_clean":"2023-10-11T11:19:01.882635+0000","last_became_active":"2023-10-11T11:15:33.659619+0000","last_became_peered":"2023-10-11T11:15:33.659619+0000","last_unstale":"2023-10-11T11:19:01.882635+0000","last_undegraded":"2023-10-11T11:19:01.882635+0000","last_fullsized":"2023-10-11T11:19:01.882635+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T17:21:16.866392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":517,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-10-11T11:19:02.775128+0000","last_change":"2023-10-11T11:15:31.670813+0000","last_active":"2023-10-11T11:19:02.775128+0000","last_peered":"2023-10-11T11:19:02.775128+0000","last_clean":"2023-10-11T11:15:27.638219+0000","last_became_active":"2023-10-11T11:15:31.670813+0000","last_became_peered":"2023-10-11T11:15:31.670813+0000","last_unstale":"2023-10-11T11:19:02.775128+0000","last_undegraded":"2023-10-11T11:19:02.775128+0000","last_fullsized":"2023-10-11T11:15:31.658785+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:41:50.872346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":516,"reported_epoch":499,"state":"active+undersized","last_fresh":"2023-10-11T11:19:00.631668+0000","last_change":"2023-10-11T11:15:31.666378+0000","last_active":"2023-10-11T11:19:00.631668+0000","last_peered":"2023-10-11T11:19:00.631668+0000","last_clean":"2023-10-11T11:15:27.638303+0000","last_became_active":"2023-10-11T11:15:31.666378+0000","last_became_peered":"2023-10-11T11:15:31.666378+0000","last_unstale":"2023-10-11T11:19:00.631668+0000","last_undegraded":"2023-10-11T11:19:00.631668+0000","last_fullsized":"2023-10-11T11:15:31.649186+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T15:35:19.331209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":516,"reported_epoch":499,"state":"active+undersized","last_fresh":"2023-10-11T11:19:00.636960+0000","last_change":"2023-10-11T11:15:31.679465+0000","last_active":"2023-10-11T11:19:00.636960+0000","last_peered":"2023-10-11T11:19:00.636960+0000","last_clean":"2023-10-11T11:15:27.638749+0000","last_became_active":"2023-10-11T11:15:31.679465+0000","last_became_peered":"2023-10-11T11:15:31.679465+0000","last_unstale":"2023-10-11T11:19:00.636960+0000","last_undegraded":"2023-10-11T11:19:00.636960+0000","last_fullsized":"2023-10-11T11:15:31.659135+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T16:03:42.656692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":11,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738804,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":785348,"kb_used_data":5812,"kb_used_omap":96,"kb_used_meta":779423,"kb_avail":93586492,"statfs":{"total":96636764160,"available":95832567808,"internally_reserved":0,"allocated":5951488,"data_stored":3106918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":98609,"internal_metadata":798129871},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 11:14:37 2023","interfaces":[{"interface":"back","average":{"1min":0.314,"5min":0.316,"15min":0.443},"min":{"1min":0.24399999999999999,"5min":0.14799999999999999,"15min":0.14599999999999999},"max":{"1min":0.48499999999999999,"5min":1.712,"15min":9.6400000000000006},"last":0.31},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.35499999999999998,"15min":0.41299999999999998},"min":{"1min":0.20899999999999999,"5min":0.17199999999999999,"15min":0.16400000000000001},"max":{"1min":3.8180000000000001,"5min":3.8180000000000001,"15min":6.3200000000000003},"last":0.26500000000000001}]},{"osd":1,"last update":"Wed Oct 11 11:18:39 2023","interfaces":[{"interface":"back","average":{"1min":0.27500000000000002,"5min":0.29299999999999998,"15min":0.32400000000000001},"min":{"1min":0.154,"5min":0.154,"15min":0.154},"max":{"1min":0.41999999999999998,"5min":0.59299999999999997,"15min":2.9540000000000002},"last":0.245},{"interface":"front","average":{"1min":0.307,"5min":0.35899999999999999,"15min":0.36499999999999999},"min":{"1min":0.18099999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.45100000000000001,"5min":3.2719999999999998,"15min":3.2719999999999998},"last":0.28299999999999997}]}]},{"osd":0,"up_from":8,"seq":34359738756,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1484496,"kb_used_data":406792,"kb_used_omap":107,"kb_used_meta":1077588,"kb_avail":92887344,"statfs":{"total":96636764160,"available":95116640256,"internally_reserved":0,"allocated":416555008,"data_stored":413648921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110309,"internal_metadata":1103450395},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Oct 11 11:14:30 2023","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.35399999999999998,"15min":0.40600000000000003},"min":{"1min":0.18099999999999999,"5min":0.155,"15min":0.14899999999999999},"max":{"1min":1.4319999999999999,"5min":2.3460000000000001,"15min":6.593},"last":0.26200000000000001},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.39100000000000001,"15min":0.48199999999999998},"min":{"1min":0.189,"5min":0.189,"15min":0.183},"max":{"1min":2.2120000000000002,"5min":2.4089999999999998,"15min":6.5960000000000001},"last":0.316}]},{"osd":2,"last update":"Wed Oct 11 11:14:30 2023","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.35499999999999998,"15min":0.47099999999999997},"min":{"1min":0.20300000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":1.2490000000000001,"5min":2.665,"15min":8.4730000000000008},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.375,"15min":0.442},"min":{"1min":0.23200000000000001,"5min":0.20499999999999999,"15min":0.19600000000000001},"max":{"1min":1.28,"5min":2.6230000000000002,"15min":7.0129999999999999},"last":0.33600000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738805,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":826440,"kb_used_data":6392,"kb_used_omap":88,"kb_used_meta":819943,"kb_avail":93545400,"statfs":{"total":96636764160,"available":95790489600,"internally_reserved":0,"allocated":6545408,"data_stored":3697286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":90523,"internal_metadata":839622245},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 11:14:33 2023","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.42799999999999999,"15min":0.41099999999999998},"min":{"1min":0.151,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":1.2809999999999999,"5min":3.5739999999999998,"15min":8.2479999999999993},"last":0.223},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.48299999999999998,"15min":0.49199999999999999},"min":{"1min":0.20699999999999999,"5min":0.182,"15min":0.17999999999999999},"max":{"1min":1.323,"5min":3.6080000000000001,"15min":8.2870000000000008},"last":0.28399999999999997}]},{"osd":2,"last update":"Wed Oct 11 11:18:40 2023","interfaces":[{"interface":"back","average":{"1min":0.33400000000000002,"5min":0.378,"15min":0.35599999999999998},"min":{"1min":0.16500000000000001,"5min":0.14699999999999999,"15min":0.14699999999999999},"max":{"1min":1.244,"5min":7.4829999999999997,"15min":10.617000000000001},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.34599999999999997,"5min":0.39000000000000001,"15min":0.34999999999999998},"min":{"1min":0.20399999999999999,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.81100000000000005,"5min":7.5350000000000001,"15min":7.5350000000000001},"last":0.49099999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-11T11:19:44.016 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-10-11T11:19:44.344 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T11:19:44.346 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2023-10-11T11:19:44.357 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":1582,"stamp":"2023-10-11T11:19:42.653748+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":11,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":107,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3096284,"kb_used_data":418996,"kb_used_omap":292,"kb_used_meta":2676955,"kb_avail":280019236,"statfs":{"total":289910292480,"available":286739697664,"internally_reserved":0,"allocated":429051904,"data_stored":420453125,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299441,"internal_metadata":2741202511},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002832"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":517,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-10-11T11:19:02.772828+0000","last_change":"2023-10-11T11:15:31.661747+0000","last_active":"2023-10-11T11:19:02.772828+0000","last_peered":"2023-10-11T11:19:02.772828+0000","last_clean":"2023-10-11T11:15:27.638215+0000","last_became_active":"2023-10-11T11:15:31.661747+0000","last_became_peered":"2023-10-11T11:15:31.661747+0000","last_unstale":"2023-10-11T11:19:02.772828+0000","last_undegraded":"2023-10-11T11:19:02.772828+0000","last_fullsized":"2023-10-11T11:15:31.649336+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:15:25.047039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":28,"reported_epoch":499,"state":"active+undersized","last_fresh":"2023-10-11T11:19:00.628569+0000","last_change":"2023-10-11T11:15:31.671363+0000","last_active":"2023-10-11T11:19:00.628569+0000","last_peered":"2023-10-11T11:19:00.628569+0000","last_clean":"2023-10-11T10:43:40.751300+0000","last_became_active":"2023-10-11T11:15:31.671363+0000","last_became_peered":"2023-10-11T11:15:31.671363+0000","last_unstale":"2023-10-11T11:19:00.628569+0000","last_undegraded":"2023-10-11T11:19:00.628569+0000","last_fullsized":"2023-10-11T11:15:31.652001+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:01:53.884964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":512,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-10-11T11:19:03.429640+0000","last_change":"2023-10-11T10:43:41.772781+0000","last_active":"2023-10-11T11:19:03.429640+0000","last_peered":"2023-10-11T11:19:03.429640+0000","last_clean":"2023-10-11T11:19:03.429640+0000","last_became_active":"2023-10-11T10:43:41.772443+0000","last_became_peered":"2023-10-11T10:43:41.772443+0000","last_unstale":"2023-10-11T11:19:03.429640+0000","last_undegraded":"2023-10-11T11:19:03.429640+0000","last_fullsized":"2023-10-11T11:19:03.429640+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T14:15:30.547284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":513,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-10-11T11:19:02.772211+0000","last_change":"2023-10-11T10:43:41.772615+0000","last_active":"2023-10-11T11:19:02.772211+0000","last_peered":"2023-10-11T11:19:02.772211+0000","last_clean":"2023-10-11T11:19:02.772211+0000","last_became_active":"2023-10-11T10:43:41.772290+0000","last_became_peered":"2023-10-11T10:43:41.772290+0000","last_unstale":"2023-10-11T11:19:02.772211+0000","last_undegraded":"2023-10-11T11:19:02.772211+0000","last_fullsized":"2023-10-11T11:19:02.772211+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:26:06.909963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":647,"reported_epoch":499,"state":"active+undersized+degraded","last_fresh":"2023-10-11T11:19:00.625113+0000","last_change":"2023-10-11T11:15:31.654022+0000","last_active":"2023-10-11T11:19:00.625113+0000","last_peered":"2023-10-11T11:19:00.625113+0000","last_clean":"2023-10-11T11:15:27.636884+0000","last_became_active":"2023-10-11T11:15:31.654022+0000","last_became_peered":"2023-10-11T11:15:31.654022+0000","last_unstale":"2023-10-11T11:19:00.625113+0000","last_undegraded":"2023-10-11T11:15:31.646946+0000","last_fullsized":"2023-10-11T11:15:31.646751+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:36.739637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:36.739637+0000","last_clean_scrub_stamp":"2023-10-11T10:43:36.739637+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T11:33:33.696447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":519,"reported_epoch":499,"state":"active+clean+remapped","last_fresh":"2023-10-11T11:19:01.882635+0000","last_change":"2023-10-11T11:15:33.660200+0000","last_active":"2023-10-11T11:19:01.882635+0000","last_peered":"2023-10-11T11:19:01.882635+0000","last_clean":"2023-10-11T11:19:01.882635+0000","last_became_active":"2023-10-11T11:15:33.659619+0000","last_became_peered":"2023-10-11T11:15:33.659619+0000","last_unstale":"2023-10-11T11:19:01.882635+0000","last_undegraded":"2023-10-11T11:19:01.882635+0000","last_fullsized":"2023-10-11T11:19:01.882635+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T17:21:16.866392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":517,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-10-11T11:19:02.775128+0000","last_change":"2023-10-11T11:15:31.670813+0000","last_active":"2023-10-11T11:19:02.775128+0000","last_peered":"2023-10-11T11:19:02.775128+0000","last_clean":"2023-10-11T11:15:27.638219+0000","last_became_active":"2023-10-11T11:15:31.670813+0000","last_became_peered":"2023-10-11T11:15:31.670813+0000","last_unstale":"2023-10-11T11:19:02.775128+0000","last_undegraded":"2023-10-11T11:19:02.775128+0000","last_fullsized":"2023-10-11T11:15:31.658785+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:41:50.872346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":516,"reported_epoch":499,"state":"active+undersized","last_fresh":"2023-10-11T11:19:00.631668+0000","last_change":"2023-10-11T11:15:31.666378+0000","last_active":"2023-10-11T11:19:00.631668+0000","last_peered":"2023-10-11T11:19:00.631668+0000","last_clean":"2023-10-11T11:15:27.638303+0000","last_became_active":"2023-10-11T11:15:31.666378+0000","last_became_peered":"2023-10-11T11:15:31.666378+0000","last_unstale":"2023-10-11T11:19:00.631668+0000","last_undegraded":"2023-10-11T11:19:00.631668+0000","last_fullsized":"2023-10-11T11:15:31.649186+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T15:35:19.331209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":516,"reported_epoch":499,"state":"active+undersized","last_fresh":"2023-10-11T11:19:00.636960+0000","last_change":"2023-10-11T11:15:31.679465+0000","last_active":"2023-10-11T11:19:00.636960+0000","last_peered":"2023-10-11T11:19:00.636960+0000","last_clean":"2023-10-11T11:15:27.638749+0000","last_became_active":"2023-10-11T11:15:31.679465+0000","last_became_peered":"2023-10-11T11:15:31.679465+0000","last_unstale":"2023-10-11T11:19:00.636960+0000","last_undegraded":"2023-10-11T11:19:00.636960+0000","last_fullsized":"2023-10-11T11:15:31.659135+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T16:03:42.656692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":11,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738804,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":785348,"kb_used_data":5812,"kb_used_omap":96,"kb_used_meta":779423,"kb_avail":93586492,"statfs":{"total":96636764160,"available":95832567808,"internally_reserved":0,"allocated":5951488,"data_stored":3106918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":98609,"internal_metadata":798129871},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 11:14:37 2023","interfaces":[{"interface":"back","average":{"1min":0.314,"5min":0.316,"15min":0.443},"min":{"1min":0.24399999999999999,"5min":0.14799999999999999,"15min":0.14599999999999999},"max":{"1min":0.48499999999999999,"5min":1.712,"15min":9.6400000000000006},"last":0.31},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.35499999999999998,"15min":0.41299999999999998},"min":{"1min":0.20899999999999999,"5min":0.17199999999999999,"15min":0.16400000000000001},"max":{"1min":3.8180000000000001,"5min":3.8180000000000001,"15min":6.3200000000000003},"last":0.26500000000000001}]},{"osd":1,"last update":"Wed Oct 11 11:18:39 2023","interfaces":[{"interface":"back","average":{"1min":0.27500000000000002,"5min":0.29299999999999998,"15min":0.32400000000000001},"min":{"1min":0.154,"5min":0.154,"15min":0.154},"max":{"1min":0.41999999999999998,"5min":0.59299999999999997,"15min":2.9540000000000002},"last":0.245},{"interface":"front","average":{"1min":0.307,"5min":0.35899999999999999,"15min":0.36499999999999999},"min":{"1min":0.18099999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.45100000000000001,"5min":3.2719999999999998,"15min":3.2719999999999998},"last":0.28299999999999997}]}]},{"osd":0,"up_from":8,"seq":34359738756,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1484496,"kb_used_data":406792,"kb_used_omap":107,"kb_used_meta":1077588,"kb_avail":92887344,"statfs":{"total":96636764160,"available":95116640256,"internally_reserved":0,"allocated":416555008,"data_stored":413648921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110309,"internal_metadata":1103450395},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Oct 11 11:14:30 2023","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.35399999999999998,"15min":0.40600000000000003},"min":{"1min":0.18099999999999999,"5min":0.155,"15min":0.14899999999999999},"max":{"1min":1.4319999999999999,"5min":2.3460000000000001,"15min":6.593},"last":0.26200000000000001},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.39100000000000001,"15min":0.48199999999999998},"min":{"1min":0.189,"5min":0.189,"15min":0.183},"max":{"1min":2.2120000000000002,"5min":2.4089999999999998,"15min":6.5960000000000001},"last":0.316}]},{"osd":2,"last update":"Wed Oct 11 11:14:30 2023","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.35499999999999998,"15min":0.47099999999999997},"min":{"1min":0.20300000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":1.2490000000000001,"5min":2.665,"15min":8.4730000000000008},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.375,"15min":0.442},"min":{"1min":0.23200000000000001,"5min":0.20499999999999999,"15min":0.19600000000000001},"max":{"1min":1.28,"5min":2.6230000000000002,"15min":7.0129999999999999},"last":0.33600000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738805,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":826440,"kb_used_data":6392,"kb_used_omap":88,"kb_used_meta":819943,"kb_avail":93545400,"statfs":{"total":96636764160,"available":95790489600,"internally_reserved":0,"allocated":6545408,"data_stored":3697286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":90523,"internal_metadata":839622245},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 11:14:33 2023","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.42799999999999999,"15min":0.41099999999999998},"min":{"1min":0.151,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":1.2809999999999999,"5min":3.5739999999999998,"15min":8.2479999999999993},"last":0.223},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.48299999999999998,"15min":0.49199999999999999},"min":{"1min":0.20699999999999999,"5min":0.182,"15min":0.17999999999999999},"max":{"1min":1.323,"5min":3.6080000000000001,"15min":8.2870000000000008},"last":0.28399999999999997}]},{"osd":2,"last update":"Wed Oct 11 11:18:40 2023","interfaces":[{"interface":"back","average":{"1min":0.33400000000000002,"5min":0.378,"15min":0.35599999999999998},"min":{"1min":0.16500000000000001,"5min":0.14699999999999999,"15min":0.14699999999999999},"max":{"1min":1.244,"5min":7.4829999999999997,"15min":10.617000000000001},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.34599999999999997,"5min":0.39000000000000001,"15min":0.34999999999999998},"min":{"1min":0.20399999999999999,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.81100000000000005,"5min":7.5350000000000001,"15min":7.5350000000000001},"last":0.49099999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-11T11:19:44.358 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-10-11T11:19:44.672 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T11:19:44.674 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2023-10-11T11:19:44.685 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":1583,"stamp":"2023-10-11T11:19:44.654134+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":11,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":107,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3096284,"kb_used_data":418996,"kb_used_omap":292,"kb_used_meta":2676955,"kb_avail":280019236,"statfs":{"total":289910292480,"available":286739697664,"internally_reserved":0,"allocated":429051904,"data_stored":420453125,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299441,"internal_metadata":2741202511},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002717"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":517,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-10-11T11:19:02.772828+0000","last_change":"2023-10-11T11:15:31.661747+0000","last_active":"2023-10-11T11:19:02.772828+0000","last_peered":"2023-10-11T11:19:02.772828+0000","last_clean":"2023-10-11T11:15:27.638215+0000","last_became_active":"2023-10-11T11:15:31.661747+0000","last_became_peered":"2023-10-11T11:15:31.661747+0000","last_unstale":"2023-10-11T11:19:02.772828+0000","last_undegraded":"2023-10-11T11:19:02.772828+0000","last_fullsized":"2023-10-11T11:15:31.649336+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:15:25.047039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":28,"reported_epoch":499,"state":"active+undersized","last_fresh":"2023-10-11T11:19:00.628569+0000","last_change":"2023-10-11T11:15:31.671363+0000","last_active":"2023-10-11T11:19:00.628569+0000","last_peered":"2023-10-11T11:19:00.628569+0000","last_clean":"2023-10-11T10:43:40.751300+0000","last_became_active":"2023-10-11T11:15:31.671363+0000","last_became_peered":"2023-10-11T11:15:31.671363+0000","last_unstale":"2023-10-11T11:19:00.628569+0000","last_undegraded":"2023-10-11T11:19:00.628569+0000","last_fullsized":"2023-10-11T11:15:31.652001+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:01:53.884964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":512,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-10-11T11:19:03.429640+0000","last_change":"2023-10-11T10:43:41.772781+0000","last_active":"2023-10-11T11:19:03.429640+0000","last_peered":"2023-10-11T11:19:03.429640+0000","last_clean":"2023-10-11T11:19:03.429640+0000","last_became_active":"2023-10-11T10:43:41.772443+0000","last_became_peered":"2023-10-11T10:43:41.772443+0000","last_unstale":"2023-10-11T11:19:03.429640+0000","last_undegraded":"2023-10-11T11:19:03.429640+0000","last_fullsized":"2023-10-11T11:19:03.429640+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T14:15:30.547284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":513,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-10-11T11:19:02.772211+0000","last_change":"2023-10-11T10:43:41.772615+0000","last_active":"2023-10-11T11:19:02.772211+0000","last_peered":"2023-10-11T11:19:02.772211+0000","last_clean":"2023-10-11T11:19:02.772211+0000","last_became_active":"2023-10-11T10:43:41.772290+0000","last_became_peered":"2023-10-11T10:43:41.772290+0000","last_unstale":"2023-10-11T11:19:02.772211+0000","last_undegraded":"2023-10-11T11:19:02.772211+0000","last_fullsized":"2023-10-11T11:19:02.772211+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:26:06.909963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":647,"reported_epoch":499,"state":"active+undersized+degraded","last_fresh":"2023-10-11T11:19:00.625113+0000","last_change":"2023-10-11T11:15:31.654022+0000","last_active":"2023-10-11T11:19:00.625113+0000","last_peered":"2023-10-11T11:19:00.625113+0000","last_clean":"2023-10-11T11:15:27.636884+0000","last_became_active":"2023-10-11T11:15:31.654022+0000","last_became_peered":"2023-10-11T11:15:31.654022+0000","last_unstale":"2023-10-11T11:19:00.625113+0000","last_undegraded":"2023-10-11T11:15:31.646946+0000","last_fullsized":"2023-10-11T11:15:31.646751+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:36.739637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:36.739637+0000","last_clean_scrub_stamp":"2023-10-11T10:43:36.739637+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T11:33:33.696447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":519,"reported_epoch":499,"state":"active+clean+remapped","last_fresh":"2023-10-11T11:19:01.882635+0000","last_change":"2023-10-11T11:15:33.660200+0000","last_active":"2023-10-11T11:19:01.882635+0000","last_peered":"2023-10-11T11:19:01.882635+0000","last_clean":"2023-10-11T11:19:01.882635+0000","last_became_active":"2023-10-11T11:15:33.659619+0000","last_became_peered":"2023-10-11T11:15:33.659619+0000","last_unstale":"2023-10-11T11:19:01.882635+0000","last_undegraded":"2023-10-11T11:19:01.882635+0000","last_fullsized":"2023-10-11T11:19:01.882635+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T17:21:16.866392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":517,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-10-11T11:19:02.775128+0000","last_change":"2023-10-11T11:15:31.670813+0000","last_active":"2023-10-11T11:19:02.775128+0000","last_peered":"2023-10-11T11:19:02.775128+0000","last_clean":"2023-10-11T11:15:27.638219+0000","last_became_active":"2023-10-11T11:15:31.670813+0000","last_became_peered":"2023-10-11T11:15:31.670813+0000","last_unstale":"2023-10-11T11:19:02.775128+0000","last_undegraded":"2023-10-11T11:19:02.775128+0000","last_fullsized":"2023-10-11T11:15:31.658785+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:41:50.872346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":516,"reported_epoch":499,"state":"active+undersized","last_fresh":"2023-10-11T11:19:00.631668+0000","last_change":"2023-10-11T11:15:31.666378+0000","last_active":"2023-10-11T11:19:00.631668+0000","last_peered":"2023-10-11T11:19:00.631668+0000","last_clean":"2023-10-11T11:15:27.638303+0000","last_became_active":"2023-10-11T11:15:31.666378+0000","last_became_peered":"2023-10-11T11:15:31.666378+0000","last_unstale":"2023-10-11T11:19:00.631668+0000","last_undegraded":"2023-10-11T11:19:00.631668+0000","last_fullsized":"2023-10-11T11:15:31.649186+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T15:35:19.331209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":516,"reported_epoch":499,"state":"active+undersized","last_fresh":"2023-10-11T11:19:00.636960+0000","last_change":"2023-10-11T11:15:31.679465+0000","last_active":"2023-10-11T11:19:00.636960+0000","last_peered":"2023-10-11T11:19:00.636960+0000","last_clean":"2023-10-11T11:15:27.638749+0000","last_became_active":"2023-10-11T11:15:31.679465+0000","last_became_peered":"2023-10-11T11:15:31.679465+0000","last_unstale":"2023-10-11T11:19:00.636960+0000","last_undegraded":"2023-10-11T11:19:00.636960+0000","last_fullsized":"2023-10-11T11:15:31.659135+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T16:03:42.656692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":11,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738804,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":785348,"kb_used_data":5812,"kb_used_omap":96,"kb_used_meta":779423,"kb_avail":93586492,"statfs":{"total":96636764160,"available":95832567808,"internally_reserved":0,"allocated":5951488,"data_stored":3106918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":98609,"internal_metadata":798129871},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 11:14:37 2023","interfaces":[{"interface":"back","average":{"1min":0.314,"5min":0.316,"15min":0.443},"min":{"1min":0.24399999999999999,"5min":0.14799999999999999,"15min":0.14599999999999999},"max":{"1min":0.48499999999999999,"5min":1.712,"15min":9.6400000000000006},"last":0.31},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.35499999999999998,"15min":0.41299999999999998},"min":{"1min":0.20899999999999999,"5min":0.17199999999999999,"15min":0.16400000000000001},"max":{"1min":3.8180000000000001,"5min":3.8180000000000001,"15min":6.3200000000000003},"last":0.26500000000000001}]},{"osd":1,"last update":"Wed Oct 11 11:18:39 2023","interfaces":[{"interface":"back","average":{"1min":0.27500000000000002,"5min":0.29299999999999998,"15min":0.32400000000000001},"min":{"1min":0.154,"5min":0.154,"15min":0.154},"max":{"1min":0.41999999999999998,"5min":0.59299999999999997,"15min":2.9540000000000002},"last":0.245},{"interface":"front","average":{"1min":0.307,"5min":0.35899999999999999,"15min":0.36499999999999999},"min":{"1min":0.18099999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.45100000000000001,"5min":3.2719999999999998,"15min":3.2719999999999998},"last":0.28299999999999997}]}]},{"osd":0,"up_from":8,"seq":34359738756,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1484496,"kb_used_data":406792,"kb_used_omap":107,"kb_used_meta":1077588,"kb_avail":92887344,"statfs":{"total":96636764160,"available":95116640256,"internally_reserved":0,"allocated":416555008,"data_stored":413648921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110309,"internal_metadata":1103450395},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Oct 11 11:14:30 2023","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.35399999999999998,"15min":0.40600000000000003},"min":{"1min":0.18099999999999999,"5min":0.155,"15min":0.14899999999999999},"max":{"1min":1.4319999999999999,"5min":2.3460000000000001,"15min":6.593},"last":0.26200000000000001},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.39100000000000001,"15min":0.48199999999999998},"min":{"1min":0.189,"5min":0.189,"15min":0.183},"max":{"1min":2.2120000000000002,"5min":2.4089999999999998,"15min":6.5960000000000001},"last":0.316}]},{"osd":2,"last update":"Wed Oct 11 11:14:30 2023","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.35499999999999998,"15min":0.47099999999999997},"min":{"1min":0.20300000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":1.2490000000000001,"5min":2.665,"15min":8.4730000000000008},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.375,"15min":0.442},"min":{"1min":0.23200000000000001,"5min":0.20499999999999999,"15min":0.19600000000000001},"max":{"1min":1.28,"5min":2.6230000000000002,"15min":7.0129999999999999},"last":0.33600000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738805,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":826440,"kb_used_data":6392,"kb_used_omap":88,"kb_used_meta":819943,"kb_avail":93545400,"statfs":{"total":96636764160,"available":95790489600,"internally_reserved":0,"allocated":6545408,"data_stored":3697286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":90523,"internal_metadata":839622245},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 11:14:33 2023","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.42799999999999999,"15min":0.41099999999999998},"min":{"1min":0.151,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":1.2809999999999999,"5min":3.5739999999999998,"15min":8.2479999999999993},"last":0.223},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.48299999999999998,"15min":0.49199999999999999},"min":{"1min":0.20699999999999999,"5min":0.182,"15min":0.17999999999999999},"max":{"1min":1.323,"5min":3.6080000000000001,"15min":8.2870000000000008},"last":0.28399999999999997}]},{"osd":2,"last update":"Wed Oct 11 11:18:40 2023","interfaces":[{"interface":"back","average":{"1min":0.33400000000000002,"5min":0.378,"15min":0.35599999999999998},"min":{"1min":0.16500000000000001,"5min":0.14699999999999999,"15min":0.14699999999999999},"max":{"1min":1.244,"5min":7.4829999999999997,"15min":10.617000000000001},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.34599999999999997,"5min":0.39000000000000001,"15min":0.34999999999999998},"min":{"1min":0.20399999999999999,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.81100000000000005,"5min":7.5350000000000001,"15min":7.5350000000000001},"last":0.49099999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-11T11:19:44.686 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-10-11T11:19:44.687 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 517, 'reported_epoch': 500, 'state': 'active+undersized', 'last_fresh': '2023-10-11T11:19:02.772828+0000', 'last_change': '2023-10-11T11:15:31.661747+0000', 'last_active': '2023-10-11T11:19:02.772828+0000', 'last_peered': '2023-10-11T11:19:02.772828+0000', 'last_clean': '2023-10-11T11:15:27.638215+0000', 'last_became_active': '2023-10-11T11:15:31.661747+0000', 'last_became_peered': '2023-10-11T11:15:31.661747+0000', 'last_unstale': '2023-10-11T11:19:02.772828+0000', 'last_undegraded': '2023-10-11T11:19:02.772828+0000', 'last_fullsized': '2023-10-11T11:15:31.649336+0000', 'mapping_epoch': 480, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_clean_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-12T22:15:25.047039+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-10-11T11:19:44.687 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-10-11T11:19:44.687 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 28, 'reported_epoch': 499, 'state': 'active+undersized', 'last_fresh': '2023-10-11T11:19:00.628569+0000', 'last_change': '2023-10-11T11:15:31.671363+0000', 'last_active': '2023-10-11T11:19:00.628569+0000', 'last_peered': '2023-10-11T11:19:00.628569+0000', 'last_clean': '2023-10-11T10:43:40.751300+0000', 'last_became_active': '2023-10-11T11:15:31.671363+0000', 'last_became_peered': '2023-10-11T11:15:31.671363+0000', 'last_unstale': '2023-10-11T11:19:00.628569+0000', 'last_undegraded': '2023-10-11T11:19:00.628569+0000', 'last_fullsized': '2023-10-11T11:15:31.652001+0000', 'mapping_epoch': 480, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_clean_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-12T21:01:53.884964+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-10-11T11:19:44.688 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-10-11T11:19:44.688 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "11'97", 'reported_seq': 647, 'reported_epoch': 499, 'state': 'active+undersized+degraded', 'last_fresh': '2023-10-11T11:19:00.625113+0000', 'last_change': '2023-10-11T11:15:31.654022+0000', 'last_active': '2023-10-11T11:19:00.625113+0000', 'last_peered': '2023-10-11T11:19:00.625113+0000', 'last_clean': '2023-10-11T11:15:27.636884+0000', 'last_became_active': '2023-10-11T11:15:31.654022+0000', 'last_became_peered': '2023-10-11T11:15:31.654022+0000', 'last_unstale': '2023-10-11T11:19:00.625113+0000', 'last_undegraded': '2023-10-11T11:15:31.646946+0000', 'last_fullsized': '2023-10-11T11:15:31.646751+0000', 'mapping_epoch': 480, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-11T10:43:36.739637+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-11T10:43:36.739637+0000', 'last_clean_scrub_stamp': '2023-10-11T10:43:36.739637+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-12T11:33:33.696447+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-10-11T11:19:44.688 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.0 is not active+clean 2023-10-11T11:19:44.688 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 519, 'reported_epoch': 499, 'state': 'active+clean+remapped', 'last_fresh': '2023-10-11T11:19:01.882635+0000', 'last_change': '2023-10-11T11:15:33.660200+0000', 'last_active': '2023-10-11T11:19:01.882635+0000', 'last_peered': '2023-10-11T11:19:01.882635+0000', 'last_clean': '2023-10-11T11:19:01.882635+0000', 'last_became_active': '2023-10-11T11:15:33.659619+0000', 'last_became_peered': '2023-10-11T11:15:33.659619+0000', 'last_unstale': '2023-10-11T11:19:01.882635+0000', 'last_undegraded': '2023-10-11T11:19:01.882635+0000', 'last_fullsized': '2023-10-11T11:19:01.882635+0000', 'mapping_epoch': 482, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 483, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_clean_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-12T17:21:16.866392+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2, 1], 'avail_no_missing': ['2', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-10-11T11:19:44.689 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-10-11T11:19:44.689 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 517, 'reported_epoch': 500, 'state': 'active+undersized', 'last_fresh': '2023-10-11T11:19:02.775128+0000', 'last_change': '2023-10-11T11:15:31.670813+0000', 'last_active': '2023-10-11T11:19:02.775128+0000', 'last_peered': '2023-10-11T11:19:02.775128+0000', 'last_clean': '2023-10-11T11:15:27.638219+0000', 'last_became_active': '2023-10-11T11:15:31.670813+0000', 'last_became_peered': '2023-10-11T11:15:31.670813+0000', 'last_unstale': '2023-10-11T11:19:02.775128+0000', 'last_undegraded': '2023-10-11T11:19:02.775128+0000', 'last_fullsized': '2023-10-11T11:15:31.658785+0000', 'mapping_epoch': 480, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_clean_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-12T21:41:50.872346+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-10-11T11:19:44.689 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-10-11T11:19:44.689 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 516, 'reported_epoch': 499, 'state': 'active+undersized', 'last_fresh': '2023-10-11T11:19:00.631668+0000', 'last_change': '2023-10-11T11:15:31.666378+0000', 'last_active': '2023-10-11T11:19:00.631668+0000', 'last_peered': '2023-10-11T11:19:00.631668+0000', 'last_clean': '2023-10-11T11:15:27.638303+0000', 'last_became_active': '2023-10-11T11:15:31.666378+0000', 'last_became_peered': '2023-10-11T11:15:31.666378+0000', 'last_unstale': '2023-10-11T11:19:00.631668+0000', 'last_undegraded': '2023-10-11T11:19:00.631668+0000', 'last_fullsized': '2023-10-11T11:15:31.649186+0000', 'mapping_epoch': 480, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_clean_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-12T15:35:19.331209+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-10-11T11:19:44.690 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-10-11T11:19:44.690 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 516, 'reported_epoch': 499, 'state': 'active+undersized', 'last_fresh': '2023-10-11T11:19:00.636960+0000', 'last_change': '2023-10-11T11:15:31.679465+0000', 'last_active': '2023-10-11T11:19:00.636960+0000', 'last_peered': '2023-10-11T11:19:00.636960+0000', 'last_clean': '2023-10-11T11:15:27.638749+0000', 'last_became_active': '2023-10-11T11:15:31.679465+0000', 'last_became_peered': '2023-10-11T11:15:31.679465+0000', 'last_unstale': '2023-10-11T11:19:00.636960+0000', 'last_undegraded': '2023-10-11T11:19:00.636960+0000', 'last_fullsized': '2023-10-11T11:15:31.659135+0000', 'mapping_epoch': 480, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_clean_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-12T16:03:42.656692+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-10-11T11:19:44.690 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-10-11T11:19:45.084 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T11:19:45.084 INFO:teuthology.orchestra.run.smithi111.stdout:{"fsid":"022ce6f2-9a5b-45ad-8b77-a1332d92f228","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2180,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":500,"num_osds":3,"num_up_osds":2,"osd_up_since":1697022930,"num_in_osds":3,"osd_in_since":1697021005,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":2},{"state_name":"active+undersized+degraded","count":1},{"state_name":"active+clean+remapped","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3170594816,"bytes_avail":286739697664,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":36,"modified":"2023-10-11T11:19:04.644408+0000","services":{}},"progress_events":{}} 2023-10-11T11:19:45.098 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-10-11T11:19:45.098 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-10-11T11:19:45.415 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T11:19:45.417 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2023-10-11T11:19:45.429 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":1583,"stamp":"2023-10-11T11:19:44.654134+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":11,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":107,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3096284,"kb_used_data":418996,"kb_used_omap":292,"kb_used_meta":2676955,"kb_avail":280019236,"statfs":{"total":289910292480,"available":286739697664,"internally_reserved":0,"allocated":429051904,"data_stored":420453125,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299441,"internal_metadata":2741202511},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002717"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":517,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-10-11T11:19:02.772828+0000","last_change":"2023-10-11T11:15:31.661747+0000","last_active":"2023-10-11T11:19:02.772828+0000","last_peered":"2023-10-11T11:19:02.772828+0000","last_clean":"2023-10-11T11:15:27.638215+0000","last_became_active":"2023-10-11T11:15:31.661747+0000","last_became_peered":"2023-10-11T11:15:31.661747+0000","last_unstale":"2023-10-11T11:19:02.772828+0000","last_undegraded":"2023-10-11T11:19:02.772828+0000","last_fullsized":"2023-10-11T11:15:31.649336+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:15:25.047039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":28,"reported_epoch":499,"state":"active+undersized","last_fresh":"2023-10-11T11:19:00.628569+0000","last_change":"2023-10-11T11:15:31.671363+0000","last_active":"2023-10-11T11:19:00.628569+0000","last_peered":"2023-10-11T11:19:00.628569+0000","last_clean":"2023-10-11T10:43:40.751300+0000","last_became_active":"2023-10-11T11:15:31.671363+0000","last_became_peered":"2023-10-11T11:15:31.671363+0000","last_unstale":"2023-10-11T11:19:00.628569+0000","last_undegraded":"2023-10-11T11:19:00.628569+0000","last_fullsized":"2023-10-11T11:15:31.652001+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:01:53.884964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":512,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-10-11T11:19:03.429640+0000","last_change":"2023-10-11T10:43:41.772781+0000","last_active":"2023-10-11T11:19:03.429640+0000","last_peered":"2023-10-11T11:19:03.429640+0000","last_clean":"2023-10-11T11:19:03.429640+0000","last_became_active":"2023-10-11T10:43:41.772443+0000","last_became_peered":"2023-10-11T10:43:41.772443+0000","last_unstale":"2023-10-11T11:19:03.429640+0000","last_undegraded":"2023-10-11T11:19:03.429640+0000","last_fullsized":"2023-10-11T11:19:03.429640+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T14:15:30.547284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":513,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-10-11T11:19:02.772211+0000","last_change":"2023-10-11T10:43:41.772615+0000","last_active":"2023-10-11T11:19:02.772211+0000","last_peered":"2023-10-11T11:19:02.772211+0000","last_clean":"2023-10-11T11:19:02.772211+0000","last_became_active":"2023-10-11T10:43:41.772290+0000","last_became_peered":"2023-10-11T10:43:41.772290+0000","last_unstale":"2023-10-11T11:19:02.772211+0000","last_undegraded":"2023-10-11T11:19:02.772211+0000","last_fullsized":"2023-10-11T11:19:02.772211+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:26:06.909963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":647,"reported_epoch":499,"state":"active+undersized+degraded","last_fresh":"2023-10-11T11:19:00.625113+0000","last_change":"2023-10-11T11:15:31.654022+0000","last_active":"2023-10-11T11:19:00.625113+0000","last_peered":"2023-10-11T11:19:00.625113+0000","last_clean":"2023-10-11T11:15:27.636884+0000","last_became_active":"2023-10-11T11:15:31.654022+0000","last_became_peered":"2023-10-11T11:15:31.654022+0000","last_unstale":"2023-10-11T11:19:00.625113+0000","last_undegraded":"2023-10-11T11:15:31.646946+0000","last_fullsized":"2023-10-11T11:15:31.646751+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:36.739637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:36.739637+0000","last_clean_scrub_stamp":"2023-10-11T10:43:36.739637+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T11:33:33.696447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":519,"reported_epoch":499,"state":"active+clean+remapped","last_fresh":"2023-10-11T11:19:01.882635+0000","last_change":"2023-10-11T11:15:33.660200+0000","last_active":"2023-10-11T11:19:01.882635+0000","last_peered":"2023-10-11T11:19:01.882635+0000","last_clean":"2023-10-11T11:19:01.882635+0000","last_became_active":"2023-10-11T11:15:33.659619+0000","last_became_peered":"2023-10-11T11:15:33.659619+0000","last_unstale":"2023-10-11T11:19:01.882635+0000","last_undegraded":"2023-10-11T11:19:01.882635+0000","last_fullsized":"2023-10-11T11:19:01.882635+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T17:21:16.866392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":517,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-10-11T11:19:02.775128+0000","last_change":"2023-10-11T11:15:31.670813+0000","last_active":"2023-10-11T11:19:02.775128+0000","last_peered":"2023-10-11T11:19:02.775128+0000","last_clean":"2023-10-11T11:15:27.638219+0000","last_became_active":"2023-10-11T11:15:31.670813+0000","last_became_peered":"2023-10-11T11:15:31.670813+0000","last_unstale":"2023-10-11T11:19:02.775128+0000","last_undegraded":"2023-10-11T11:19:02.775128+0000","last_fullsized":"2023-10-11T11:15:31.658785+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:41:50.872346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":516,"reported_epoch":499,"state":"active+undersized","last_fresh":"2023-10-11T11:19:00.631668+0000","last_change":"2023-10-11T11:15:31.666378+0000","last_active":"2023-10-11T11:19:00.631668+0000","last_peered":"2023-10-11T11:19:00.631668+0000","last_clean":"2023-10-11T11:15:27.638303+0000","last_became_active":"2023-10-11T11:15:31.666378+0000","last_became_peered":"2023-10-11T11:15:31.666378+0000","last_unstale":"2023-10-11T11:19:00.631668+0000","last_undegraded":"2023-10-11T11:19:00.631668+0000","last_fullsized":"2023-10-11T11:15:31.649186+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T15:35:19.331209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":516,"reported_epoch":499,"state":"active+undersized","last_fresh":"2023-10-11T11:19:00.636960+0000","last_change":"2023-10-11T11:15:31.679465+0000","last_active":"2023-10-11T11:19:00.636960+0000","last_peered":"2023-10-11T11:19:00.636960+0000","last_clean":"2023-10-11T11:15:27.638749+0000","last_became_active":"2023-10-11T11:15:31.679465+0000","last_became_peered":"2023-10-11T11:15:31.679465+0000","last_unstale":"2023-10-11T11:19:00.636960+0000","last_undegraded":"2023-10-11T11:19:00.636960+0000","last_fullsized":"2023-10-11T11:15:31.659135+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T16:03:42.656692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":11,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738804,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":785348,"kb_used_data":5812,"kb_used_omap":96,"kb_used_meta":779423,"kb_avail":93586492,"statfs":{"total":96636764160,"available":95832567808,"internally_reserved":0,"allocated":5951488,"data_stored":3106918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":98609,"internal_metadata":798129871},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 11:14:37 2023","interfaces":[{"interface":"back","average":{"1min":0.314,"5min":0.316,"15min":0.443},"min":{"1min":0.24399999999999999,"5min":0.14799999999999999,"15min":0.14599999999999999},"max":{"1min":0.48499999999999999,"5min":1.712,"15min":9.6400000000000006},"last":0.31},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.35499999999999998,"15min":0.41299999999999998},"min":{"1min":0.20899999999999999,"5min":0.17199999999999999,"15min":0.16400000000000001},"max":{"1min":3.8180000000000001,"5min":3.8180000000000001,"15min":6.3200000000000003},"last":0.26500000000000001}]},{"osd":1,"last update":"Wed Oct 11 11:18:39 2023","interfaces":[{"interface":"back","average":{"1min":0.27500000000000002,"5min":0.29299999999999998,"15min":0.32400000000000001},"min":{"1min":0.154,"5min":0.154,"15min":0.154},"max":{"1min":0.41999999999999998,"5min":0.59299999999999997,"15min":2.9540000000000002},"last":0.245},{"interface":"front","average":{"1min":0.307,"5min":0.35899999999999999,"15min":0.36499999999999999},"min":{"1min":0.18099999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.45100000000000001,"5min":3.2719999999999998,"15min":3.2719999999999998},"last":0.28299999999999997}]}]},{"osd":0,"up_from":8,"seq":34359738756,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1484496,"kb_used_data":406792,"kb_used_omap":107,"kb_used_meta":1077588,"kb_avail":92887344,"statfs":{"total":96636764160,"available":95116640256,"internally_reserved":0,"allocated":416555008,"data_stored":413648921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110309,"internal_metadata":1103450395},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Oct 11 11:14:30 2023","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.35399999999999998,"15min":0.40600000000000003},"min":{"1min":0.18099999999999999,"5min":0.155,"15min":0.14899999999999999},"max":{"1min":1.4319999999999999,"5min":2.3460000000000001,"15min":6.593},"last":0.26200000000000001},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.39100000000000001,"15min":0.48199999999999998},"min":{"1min":0.189,"5min":0.189,"15min":0.183},"max":{"1min":2.2120000000000002,"5min":2.4089999999999998,"15min":6.5960000000000001},"last":0.316}]},{"osd":2,"last update":"Wed Oct 11 11:14:30 2023","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.35499999999999998,"15min":0.47099999999999997},"min":{"1min":0.20300000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":1.2490000000000001,"5min":2.665,"15min":8.4730000000000008},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.375,"15min":0.442},"min":{"1min":0.23200000000000001,"5min":0.20499999999999999,"15min":0.19600000000000001},"max":{"1min":1.28,"5min":2.6230000000000002,"15min":7.0129999999999999},"last":0.33600000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738805,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":826440,"kb_used_data":6392,"kb_used_omap":88,"kb_used_meta":819943,"kb_avail":93545400,"statfs":{"total":96636764160,"available":95790489600,"internally_reserved":0,"allocated":6545408,"data_stored":3697286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":90523,"internal_metadata":839622245},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 11:14:33 2023","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.42799999999999999,"15min":0.41099999999999998},"min":{"1min":0.151,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":1.2809999999999999,"5min":3.5739999999999998,"15min":8.2479999999999993},"last":0.223},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.48299999999999998,"15min":0.49199999999999999},"min":{"1min":0.20699999999999999,"5min":0.182,"15min":0.17999999999999999},"max":{"1min":1.323,"5min":3.6080000000000001,"15min":8.2870000000000008},"last":0.28399999999999997}]},{"osd":2,"last update":"Wed Oct 11 11:18:40 2023","interfaces":[{"interface":"back","average":{"1min":0.33400000000000002,"5min":0.378,"15min":0.35599999999999998},"min":{"1min":0.16500000000000001,"5min":0.14699999999999999,"15min":0.14699999999999999},"max":{"1min":1.244,"5min":7.4829999999999997,"15min":10.617000000000001},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.34599999999999997,"5min":0.39000000000000001,"15min":0.34999999999999998},"min":{"1min":0.20399999999999999,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.81100000000000005,"5min":7.5350000000000001,"15min":7.5350000000000001},"last":0.49099999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-11T11:19:46.138 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~254s 2023-10-11T11:19:48.431 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-10-11T11:19:48.764 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T11:19:48.766 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2023-10-11T11:19:48.777 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":1585,"stamp":"2023-10-11T11:19:48.654884+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":11,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":107,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3096284,"kb_used_data":418996,"kb_used_omap":292,"kb_used_meta":2676955,"kb_avail":280019236,"statfs":{"total":289910292480,"available":286739697664,"internally_reserved":0,"allocated":429051904,"data_stored":420453125,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299441,"internal_metadata":2741202511},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002455"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":517,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-10-11T11:19:02.772828+0000","last_change":"2023-10-11T11:15:31.661747+0000","last_active":"2023-10-11T11:19:02.772828+0000","last_peered":"2023-10-11T11:19:02.772828+0000","last_clean":"2023-10-11T11:15:27.638215+0000","last_became_active":"2023-10-11T11:15:31.661747+0000","last_became_peered":"2023-10-11T11:15:31.661747+0000","last_unstale":"2023-10-11T11:19:02.772828+0000","last_undegraded":"2023-10-11T11:19:02.772828+0000","last_fullsized":"2023-10-11T11:15:31.649336+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:15:25.047039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":29,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-10-11T11:19:02.774213+0000","last_change":"2023-10-11T11:15:31.671363+0000","last_active":"2023-10-11T11:19:02.774213+0000","last_peered":"2023-10-11T11:19:02.774213+0000","last_clean":"2023-10-11T10:43:40.751300+0000","last_became_active":"2023-10-11T11:15:31.671363+0000","last_became_peered":"2023-10-11T11:15:31.671363+0000","last_unstale":"2023-10-11T11:19:02.774213+0000","last_undegraded":"2023-10-11T11:19:02.774213+0000","last_fullsized":"2023-10-11T11:15:31.652001+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:01:53.884964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":512,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-10-11T11:19:03.429640+0000","last_change":"2023-10-11T10:43:41.772781+0000","last_active":"2023-10-11T11:19:03.429640+0000","last_peered":"2023-10-11T11:19:03.429640+0000","last_clean":"2023-10-11T11:19:03.429640+0000","last_became_active":"2023-10-11T10:43:41.772443+0000","last_became_peered":"2023-10-11T10:43:41.772443+0000","last_unstale":"2023-10-11T11:19:03.429640+0000","last_undegraded":"2023-10-11T11:19:03.429640+0000","last_fullsized":"2023-10-11T11:19:03.429640+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T14:15:30.547284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":513,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-10-11T11:19:02.772211+0000","last_change":"2023-10-11T10:43:41.772615+0000","last_active":"2023-10-11T11:19:02.772211+0000","last_peered":"2023-10-11T11:19:02.772211+0000","last_clean":"2023-10-11T11:19:02.772211+0000","last_became_active":"2023-10-11T10:43:41.772290+0000","last_became_peered":"2023-10-11T10:43:41.772290+0000","last_unstale":"2023-10-11T11:19:02.772211+0000","last_undegraded":"2023-10-11T11:19:02.772211+0000","last_fullsized":"2023-10-11T11:19:02.772211+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:26:06.909963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":648,"reported_epoch":500,"state":"active+undersized+degraded","last_fresh":"2023-10-11T11:19:02.769176+0000","last_change":"2023-10-11T11:15:31.654022+0000","last_active":"2023-10-11T11:19:02.769176+0000","last_peered":"2023-10-11T11:19:02.769176+0000","last_clean":"2023-10-11T11:15:27.636884+0000","last_became_active":"2023-10-11T11:15:31.654022+0000","last_became_peered":"2023-10-11T11:15:31.654022+0000","last_unstale":"2023-10-11T11:19:02.769176+0000","last_undegraded":"2023-10-11T11:15:31.646946+0000","last_fullsized":"2023-10-11T11:15:31.646751+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:36.739637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:36.739637+0000","last_clean_scrub_stamp":"2023-10-11T10:43:36.739637+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T11:33:33.696447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":519,"reported_epoch":499,"state":"active+clean+remapped","last_fresh":"2023-10-11T11:19:01.882635+0000","last_change":"2023-10-11T11:15:33.660200+0000","last_active":"2023-10-11T11:19:01.882635+0000","last_peered":"2023-10-11T11:19:01.882635+0000","last_clean":"2023-10-11T11:19:01.882635+0000","last_became_active":"2023-10-11T11:15:33.659619+0000","last_became_peered":"2023-10-11T11:15:33.659619+0000","last_unstale":"2023-10-11T11:19:01.882635+0000","last_undegraded":"2023-10-11T11:19:01.882635+0000","last_fullsized":"2023-10-11T11:19:01.882635+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T17:21:16.866392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":517,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-10-11T11:19:02.775128+0000","last_change":"2023-10-11T11:15:31.670813+0000","last_active":"2023-10-11T11:19:02.775128+0000","last_peered":"2023-10-11T11:19:02.775128+0000","last_clean":"2023-10-11T11:15:27.638219+0000","last_became_active":"2023-10-11T11:15:31.670813+0000","last_became_peered":"2023-10-11T11:15:31.670813+0000","last_unstale":"2023-10-11T11:19:02.775128+0000","last_undegraded":"2023-10-11T11:19:02.775128+0000","last_fullsized":"2023-10-11T11:15:31.658785+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:41:50.872346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":516,"reported_epoch":499,"state":"active+undersized","last_fresh":"2023-10-11T11:19:00.631668+0000","last_change":"2023-10-11T11:15:31.666378+0000","last_active":"2023-10-11T11:19:00.631668+0000","last_peered":"2023-10-11T11:19:00.631668+0000","last_clean":"2023-10-11T11:15:27.638303+0000","last_became_active":"2023-10-11T11:15:31.666378+0000","last_became_peered":"2023-10-11T11:15:31.666378+0000","last_unstale":"2023-10-11T11:19:00.631668+0000","last_undegraded":"2023-10-11T11:19:00.631668+0000","last_fullsized":"2023-10-11T11:15:31.649186+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T15:35:19.331209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":516,"reported_epoch":499,"state":"active+undersized","last_fresh":"2023-10-11T11:19:00.636960+0000","last_change":"2023-10-11T11:15:31.679465+0000","last_active":"2023-10-11T11:19:00.636960+0000","last_peered":"2023-10-11T11:19:00.636960+0000","last_clean":"2023-10-11T11:15:27.638749+0000","last_became_active":"2023-10-11T11:15:31.679465+0000","last_became_peered":"2023-10-11T11:15:31.679465+0000","last_unstale":"2023-10-11T11:19:00.636960+0000","last_undegraded":"2023-10-11T11:19:00.636960+0000","last_fullsized":"2023-10-11T11:15:31.659135+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T16:03:42.656692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":11,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738805,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":785348,"kb_used_data":5812,"kb_used_omap":96,"kb_used_meta":779423,"kb_avail":93586492,"statfs":{"total":96636764160,"available":95832567808,"internally_reserved":0,"allocated":5951488,"data_stored":3106918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":98609,"internal_metadata":798129871},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 11:14:37 2023","interfaces":[{"interface":"back","average":{"1min":0.314,"5min":0.316,"15min":0.443},"min":{"1min":0.24399999999999999,"5min":0.14799999999999999,"15min":0.14599999999999999},"max":{"1min":0.48499999999999999,"5min":1.712,"15min":9.6400000000000006},"last":0.31},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.35499999999999998,"15min":0.41299999999999998},"min":{"1min":0.20899999999999999,"5min":0.17199999999999999,"15min":0.16400000000000001},"max":{"1min":3.8180000000000001,"5min":3.8180000000000001,"15min":6.3200000000000003},"last":0.26500000000000001}]},{"osd":1,"last update":"Wed Oct 11 11:19:40 2023","interfaces":[{"interface":"back","average":{"1min":0.29599999999999999,"5min":0.28499999999999998,"15min":0.32300000000000001},"min":{"1min":0.17699999999999999,"5min":0.154,"15min":0.154},"max":{"1min":0.40600000000000003,"5min":0.41999999999999998,"15min":2.9540000000000002},"last":0.307},{"interface":"front","average":{"1min":0.317,"5min":0.31,"15min":0.36199999999999999},"min":{"1min":0.20899999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.45000000000000001,"5min":0.49099999999999999,"15min":3.2719999999999998},"last":0.35199999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738756,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1484496,"kb_used_data":406792,"kb_used_omap":107,"kb_used_meta":1077588,"kb_avail":92887344,"statfs":{"total":96636764160,"available":95116640256,"internally_reserved":0,"allocated":416555008,"data_stored":413648921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110309,"internal_metadata":1103450395},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Oct 11 11:14:30 2023","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.35399999999999998,"15min":0.40600000000000003},"min":{"1min":0.18099999999999999,"5min":0.155,"15min":0.14899999999999999},"max":{"1min":1.4319999999999999,"5min":2.3460000000000001,"15min":6.593},"last":0.26200000000000001},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.39100000000000001,"15min":0.48199999999999998},"min":{"1min":0.189,"5min":0.189,"15min":0.183},"max":{"1min":2.2120000000000002,"5min":2.4089999999999998,"15min":6.5960000000000001},"last":0.316}]},{"osd":2,"last update":"Wed Oct 11 11:14:30 2023","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.35499999999999998,"15min":0.47099999999999997},"min":{"1min":0.20300000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":1.2490000000000001,"5min":2.665,"15min":8.4730000000000008},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.375,"15min":0.442},"min":{"1min":0.23200000000000001,"5min":0.20499999999999999,"15min":0.19600000000000001},"max":{"1min":1.28,"5min":2.6230000000000002,"15min":7.0129999999999999},"last":0.33600000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738806,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":826440,"kb_used_data":6392,"kb_used_omap":88,"kb_used_meta":819943,"kb_avail":93545400,"statfs":{"total":96636764160,"available":95790489600,"internally_reserved":0,"allocated":6545408,"data_stored":3697286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":90523,"internal_metadata":839622245},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 11:14:33 2023","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.42799999999999999,"15min":0.41099999999999998},"min":{"1min":0.151,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":1.2809999999999999,"5min":3.5739999999999998,"15min":8.2479999999999993},"last":0.223},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.48299999999999998,"15min":0.49199999999999999},"min":{"1min":0.20699999999999999,"5min":0.182,"15min":0.17999999999999999},"max":{"1min":1.323,"5min":3.6080000000000001,"15min":8.2870000000000008},"last":0.28399999999999997}]},{"osd":2,"last update":"Wed Oct 11 11:19:41 2023","interfaces":[{"interface":"back","average":{"1min":0.314,"5min":0.35399999999999998,"15min":0.35899999999999999},"min":{"1min":0.186,"5min":0.14699999999999999,"15min":0.14699999999999999},"max":{"1min":0.45500000000000002,"5min":7.4829999999999997,"15min":10.617000000000001},"last":0.27600000000000002},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.39400000000000002,"15min":0.35099999999999998},"min":{"1min":0.21199999999999999,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.49099999999999999,"5min":7.5350000000000001,"15min":7.5350000000000001},"last":0.30499999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-11T11:19:48.779 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-10-11T11:19:49.095 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T11:19:49.097 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2023-10-11T11:19:49.109 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":1585,"stamp":"2023-10-11T11:19:48.654884+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":11,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":107,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3096284,"kb_used_data":418996,"kb_used_omap":292,"kb_used_meta":2676955,"kb_avail":280019236,"statfs":{"total":289910292480,"available":286739697664,"internally_reserved":0,"allocated":429051904,"data_stored":420453125,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299441,"internal_metadata":2741202511},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002455"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":517,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-10-11T11:19:02.772828+0000","last_change":"2023-10-11T11:15:31.661747+0000","last_active":"2023-10-11T11:19:02.772828+0000","last_peered":"2023-10-11T11:19:02.772828+0000","last_clean":"2023-10-11T11:15:27.638215+0000","last_became_active":"2023-10-11T11:15:31.661747+0000","last_became_peered":"2023-10-11T11:15:31.661747+0000","last_unstale":"2023-10-11T11:19:02.772828+0000","last_undegraded":"2023-10-11T11:19:02.772828+0000","last_fullsized":"2023-10-11T11:15:31.649336+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:15:25.047039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":29,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-10-11T11:19:02.774213+0000","last_change":"2023-10-11T11:15:31.671363+0000","last_active":"2023-10-11T11:19:02.774213+0000","last_peered":"2023-10-11T11:19:02.774213+0000","last_clean":"2023-10-11T10:43:40.751300+0000","last_became_active":"2023-10-11T11:15:31.671363+0000","last_became_peered":"2023-10-11T11:15:31.671363+0000","last_unstale":"2023-10-11T11:19:02.774213+0000","last_undegraded":"2023-10-11T11:19:02.774213+0000","last_fullsized":"2023-10-11T11:15:31.652001+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:01:53.884964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":512,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-10-11T11:19:03.429640+0000","last_change":"2023-10-11T10:43:41.772781+0000","last_active":"2023-10-11T11:19:03.429640+0000","last_peered":"2023-10-11T11:19:03.429640+0000","last_clean":"2023-10-11T11:19:03.429640+0000","last_became_active":"2023-10-11T10:43:41.772443+0000","last_became_peered":"2023-10-11T10:43:41.772443+0000","last_unstale":"2023-10-11T11:19:03.429640+0000","last_undegraded":"2023-10-11T11:19:03.429640+0000","last_fullsized":"2023-10-11T11:19:03.429640+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T14:15:30.547284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":513,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-10-11T11:19:02.772211+0000","last_change":"2023-10-11T10:43:41.772615+0000","last_active":"2023-10-11T11:19:02.772211+0000","last_peered":"2023-10-11T11:19:02.772211+0000","last_clean":"2023-10-11T11:19:02.772211+0000","last_became_active":"2023-10-11T10:43:41.772290+0000","last_became_peered":"2023-10-11T10:43:41.772290+0000","last_unstale":"2023-10-11T11:19:02.772211+0000","last_undegraded":"2023-10-11T11:19:02.772211+0000","last_fullsized":"2023-10-11T11:19:02.772211+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:26:06.909963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":648,"reported_epoch":500,"state":"active+undersized+degraded","last_fresh":"2023-10-11T11:19:02.769176+0000","last_change":"2023-10-11T11:15:31.654022+0000","last_active":"2023-10-11T11:19:02.769176+0000","last_peered":"2023-10-11T11:19:02.769176+0000","last_clean":"2023-10-11T11:15:27.636884+0000","last_became_active":"2023-10-11T11:15:31.654022+0000","last_became_peered":"2023-10-11T11:15:31.654022+0000","last_unstale":"2023-10-11T11:19:02.769176+0000","last_undegraded":"2023-10-11T11:15:31.646946+0000","last_fullsized":"2023-10-11T11:15:31.646751+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:36.739637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:36.739637+0000","last_clean_scrub_stamp":"2023-10-11T10:43:36.739637+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T11:33:33.696447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":519,"reported_epoch":499,"state":"active+clean+remapped","last_fresh":"2023-10-11T11:19:01.882635+0000","last_change":"2023-10-11T11:15:33.660200+0000","last_active":"2023-10-11T11:19:01.882635+0000","last_peered":"2023-10-11T11:19:01.882635+0000","last_clean":"2023-10-11T11:19:01.882635+0000","last_became_active":"2023-10-11T11:15:33.659619+0000","last_became_peered":"2023-10-11T11:15:33.659619+0000","last_unstale":"2023-10-11T11:19:01.882635+0000","last_undegraded":"2023-10-11T11:19:01.882635+0000","last_fullsized":"2023-10-11T11:19:01.882635+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T17:21:16.866392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":517,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-10-11T11:19:02.775128+0000","last_change":"2023-10-11T11:15:31.670813+0000","last_active":"2023-10-11T11:19:02.775128+0000","last_peered":"2023-10-11T11:19:02.775128+0000","last_clean":"2023-10-11T11:15:27.638219+0000","last_became_active":"2023-10-11T11:15:31.670813+0000","last_became_peered":"2023-10-11T11:15:31.670813+0000","last_unstale":"2023-10-11T11:19:02.775128+0000","last_undegraded":"2023-10-11T11:19:02.775128+0000","last_fullsized":"2023-10-11T11:15:31.658785+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:41:50.872346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":516,"reported_epoch":499,"state":"active+undersized","last_fresh":"2023-10-11T11:19:00.631668+0000","last_change":"2023-10-11T11:15:31.666378+0000","last_active":"2023-10-11T11:19:00.631668+0000","last_peered":"2023-10-11T11:19:00.631668+0000","last_clean":"2023-10-11T11:15:27.638303+0000","last_became_active":"2023-10-11T11:15:31.666378+0000","last_became_peered":"2023-10-11T11:15:31.666378+0000","last_unstale":"2023-10-11T11:19:00.631668+0000","last_undegraded":"2023-10-11T11:19:00.631668+0000","last_fullsized":"2023-10-11T11:15:31.649186+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T15:35:19.331209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":516,"reported_epoch":499,"state":"active+undersized","last_fresh":"2023-10-11T11:19:00.636960+0000","last_change":"2023-10-11T11:15:31.679465+0000","last_active":"2023-10-11T11:19:00.636960+0000","last_peered":"2023-10-11T11:19:00.636960+0000","last_clean":"2023-10-11T11:15:27.638749+0000","last_became_active":"2023-10-11T11:15:31.679465+0000","last_became_peered":"2023-10-11T11:15:31.679465+0000","last_unstale":"2023-10-11T11:19:00.636960+0000","last_undegraded":"2023-10-11T11:19:00.636960+0000","last_fullsized":"2023-10-11T11:15:31.659135+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T16:03:42.656692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":11,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738805,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":785348,"kb_used_data":5812,"kb_used_omap":96,"kb_used_meta":779423,"kb_avail":93586492,"statfs":{"total":96636764160,"available":95832567808,"internally_reserved":0,"allocated":5951488,"data_stored":3106918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":98609,"internal_metadata":798129871},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 11:14:37 2023","interfaces":[{"interface":"back","average":{"1min":0.314,"5min":0.316,"15min":0.443},"min":{"1min":0.24399999999999999,"5min":0.14799999999999999,"15min":0.14599999999999999},"max":{"1min":0.48499999999999999,"5min":1.712,"15min":9.6400000000000006},"last":0.31},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.35499999999999998,"15min":0.41299999999999998},"min":{"1min":0.20899999999999999,"5min":0.17199999999999999,"15min":0.16400000000000001},"max":{"1min":3.8180000000000001,"5min":3.8180000000000001,"15min":6.3200000000000003},"last":0.26500000000000001}]},{"osd":1,"last update":"Wed Oct 11 11:19:40 2023","interfaces":[{"interface":"back","average":{"1min":0.29599999999999999,"5min":0.28499999999999998,"15min":0.32300000000000001},"min":{"1min":0.17699999999999999,"5min":0.154,"15min":0.154},"max":{"1min":0.40600000000000003,"5min":0.41999999999999998,"15min":2.9540000000000002},"last":0.307},{"interface":"front","average":{"1min":0.317,"5min":0.31,"15min":0.36199999999999999},"min":{"1min":0.20899999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.45000000000000001,"5min":0.49099999999999999,"15min":3.2719999999999998},"last":0.35199999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738756,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1484496,"kb_used_data":406792,"kb_used_omap":107,"kb_used_meta":1077588,"kb_avail":92887344,"statfs":{"total":96636764160,"available":95116640256,"internally_reserved":0,"allocated":416555008,"data_stored":413648921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110309,"internal_metadata":1103450395},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Oct 11 11:14:30 2023","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.35399999999999998,"15min":0.40600000000000003},"min":{"1min":0.18099999999999999,"5min":0.155,"15min":0.14899999999999999},"max":{"1min":1.4319999999999999,"5min":2.3460000000000001,"15min":6.593},"last":0.26200000000000001},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.39100000000000001,"15min":0.48199999999999998},"min":{"1min":0.189,"5min":0.189,"15min":0.183},"max":{"1min":2.2120000000000002,"5min":2.4089999999999998,"15min":6.5960000000000001},"last":0.316}]},{"osd":2,"last update":"Wed Oct 11 11:14:30 2023","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.35499999999999998,"15min":0.47099999999999997},"min":{"1min":0.20300000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":1.2490000000000001,"5min":2.665,"15min":8.4730000000000008},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.375,"15min":0.442},"min":{"1min":0.23200000000000001,"5min":0.20499999999999999,"15min":0.19600000000000001},"max":{"1min":1.28,"5min":2.6230000000000002,"15min":7.0129999999999999},"last":0.33600000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738806,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":826440,"kb_used_data":6392,"kb_used_omap":88,"kb_used_meta":819943,"kb_avail":93545400,"statfs":{"total":96636764160,"available":95790489600,"internally_reserved":0,"allocated":6545408,"data_stored":3697286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":90523,"internal_metadata":839622245},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 11:14:33 2023","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.42799999999999999,"15min":0.41099999999999998},"min":{"1min":0.151,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":1.2809999999999999,"5min":3.5739999999999998,"15min":8.2479999999999993},"last":0.223},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.48299999999999998,"15min":0.49199999999999999},"min":{"1min":0.20699999999999999,"5min":0.182,"15min":0.17999999999999999},"max":{"1min":1.323,"5min":3.6080000000000001,"15min":8.2870000000000008},"last":0.28399999999999997}]},{"osd":2,"last update":"Wed Oct 11 11:19:41 2023","interfaces":[{"interface":"back","average":{"1min":0.314,"5min":0.35399999999999998,"15min":0.35899999999999999},"min":{"1min":0.186,"5min":0.14699999999999999,"15min":0.14699999999999999},"max":{"1min":0.45500000000000002,"5min":7.4829999999999997,"15min":10.617000000000001},"last":0.27600000000000002},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.39400000000000002,"15min":0.35099999999999998},"min":{"1min":0.21199999999999999,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.49099999999999999,"5min":7.5350000000000001,"15min":7.5350000000000001},"last":0.30499999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-11T11:19:49.110 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-10-11T11:19:49.110 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 517, 'reported_epoch': 500, 'state': 'active+undersized', 'last_fresh': '2023-10-11T11:19:02.772828+0000', 'last_change': '2023-10-11T11:15:31.661747+0000', 'last_active': '2023-10-11T11:19:02.772828+0000', 'last_peered': '2023-10-11T11:19:02.772828+0000', 'last_clean': '2023-10-11T11:15:27.638215+0000', 'last_became_active': '2023-10-11T11:15:31.661747+0000', 'last_became_peered': '2023-10-11T11:15:31.661747+0000', 'last_unstale': '2023-10-11T11:19:02.772828+0000', 'last_undegraded': '2023-10-11T11:19:02.772828+0000', 'last_fullsized': '2023-10-11T11:15:31.649336+0000', 'mapping_epoch': 480, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_clean_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-12T22:15:25.047039+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-10-11T11:19:49.111 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-10-11T11:19:49.111 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 29, 'reported_epoch': 500, 'state': 'active+undersized', 'last_fresh': '2023-10-11T11:19:02.774213+0000', 'last_change': '2023-10-11T11:15:31.671363+0000', 'last_active': '2023-10-11T11:19:02.774213+0000', 'last_peered': '2023-10-11T11:19:02.774213+0000', 'last_clean': '2023-10-11T10:43:40.751300+0000', 'last_became_active': '2023-10-11T11:15:31.671363+0000', 'last_became_peered': '2023-10-11T11:15:31.671363+0000', 'last_unstale': '2023-10-11T11:19:02.774213+0000', 'last_undegraded': '2023-10-11T11:19:02.774213+0000', 'last_fullsized': '2023-10-11T11:15:31.652001+0000', 'mapping_epoch': 480, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_clean_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-12T21:01:53.884964+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-10-11T11:19:49.111 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-10-11T11:19:49.111 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "11'97", 'reported_seq': 648, 'reported_epoch': 500, 'state': 'active+undersized+degraded', 'last_fresh': '2023-10-11T11:19:02.769176+0000', 'last_change': '2023-10-11T11:15:31.654022+0000', 'last_active': '2023-10-11T11:19:02.769176+0000', 'last_peered': '2023-10-11T11:19:02.769176+0000', 'last_clean': '2023-10-11T11:15:27.636884+0000', 'last_became_active': '2023-10-11T11:15:31.654022+0000', 'last_became_peered': '2023-10-11T11:15:31.654022+0000', 'last_unstale': '2023-10-11T11:19:02.769176+0000', 'last_undegraded': '2023-10-11T11:15:31.646946+0000', 'last_fullsized': '2023-10-11T11:15:31.646751+0000', 'mapping_epoch': 480, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-11T10:43:36.739637+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-11T10:43:36.739637+0000', 'last_clean_scrub_stamp': '2023-10-11T10:43:36.739637+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-12T11:33:33.696447+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-10-11T11:19:49.112 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.0 is not active+clean 2023-10-11T11:19:49.112 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 519, 'reported_epoch': 499, 'state': 'active+clean+remapped', 'last_fresh': '2023-10-11T11:19:01.882635+0000', 'last_change': '2023-10-11T11:15:33.660200+0000', 'last_active': '2023-10-11T11:19:01.882635+0000', 'last_peered': '2023-10-11T11:19:01.882635+0000', 'last_clean': '2023-10-11T11:19:01.882635+0000', 'last_became_active': '2023-10-11T11:15:33.659619+0000', 'last_became_peered': '2023-10-11T11:15:33.659619+0000', 'last_unstale': '2023-10-11T11:19:01.882635+0000', 'last_undegraded': '2023-10-11T11:19:01.882635+0000', 'last_fullsized': '2023-10-11T11:19:01.882635+0000', 'mapping_epoch': 482, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 483, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_clean_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-12T17:21:16.866392+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2, 1], 'avail_no_missing': ['2', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-10-11T11:19:49.112 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-10-11T11:19:49.113 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 517, 'reported_epoch': 500, 'state': 'active+undersized', 'last_fresh': '2023-10-11T11:19:02.775128+0000', 'last_change': '2023-10-11T11:15:31.670813+0000', 'last_active': '2023-10-11T11:19:02.775128+0000', 'last_peered': '2023-10-11T11:19:02.775128+0000', 'last_clean': '2023-10-11T11:15:27.638219+0000', 'last_became_active': '2023-10-11T11:15:31.670813+0000', 'last_became_peered': '2023-10-11T11:15:31.670813+0000', 'last_unstale': '2023-10-11T11:19:02.775128+0000', 'last_undegraded': '2023-10-11T11:19:02.775128+0000', 'last_fullsized': '2023-10-11T11:15:31.658785+0000', 'mapping_epoch': 480, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_clean_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-12T21:41:50.872346+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-10-11T11:19:49.113 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-10-11T11:19:49.113 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 516, 'reported_epoch': 499, 'state': 'active+undersized', 'last_fresh': '2023-10-11T11:19:00.631668+0000', 'last_change': '2023-10-11T11:15:31.666378+0000', 'last_active': '2023-10-11T11:19:00.631668+0000', 'last_peered': '2023-10-11T11:19:00.631668+0000', 'last_clean': '2023-10-11T11:15:27.638303+0000', 'last_became_active': '2023-10-11T11:15:31.666378+0000', 'last_became_peered': '2023-10-11T11:15:31.666378+0000', 'last_unstale': '2023-10-11T11:19:00.631668+0000', 'last_undegraded': '2023-10-11T11:19:00.631668+0000', 'last_fullsized': '2023-10-11T11:15:31.649186+0000', 'mapping_epoch': 480, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_clean_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-12T15:35:19.331209+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-10-11T11:19:49.114 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-10-11T11:19:49.114 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 516, 'reported_epoch': 499, 'state': 'active+undersized', 'last_fresh': '2023-10-11T11:19:00.636960+0000', 'last_change': '2023-10-11T11:15:31.679465+0000', 'last_active': '2023-10-11T11:19:00.636960+0000', 'last_peered': '2023-10-11T11:19:00.636960+0000', 'last_clean': '2023-10-11T11:15:27.638749+0000', 'last_became_active': '2023-10-11T11:15:31.679465+0000', 'last_became_peered': '2023-10-11T11:15:31.679465+0000', 'last_unstale': '2023-10-11T11:19:00.636960+0000', 'last_undegraded': '2023-10-11T11:19:00.636960+0000', 'last_fullsized': '2023-10-11T11:15:31.659135+0000', 'mapping_epoch': 480, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_clean_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-12T16:03:42.656692+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-10-11T11:19:49.115 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-10-11T11:19:49.512 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T11:19:49.513 INFO:teuthology.orchestra.run.smithi111.stdout:{"fsid":"022ce6f2-9a5b-45ad-8b77-a1332d92f228","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2184,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":500,"num_osds":3,"num_up_osds":2,"osd_up_since":1697022930,"num_in_osds":3,"osd_in_since":1697021005,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":2},{"state_name":"active+undersized+degraded","count":1},{"state_name":"active+clean+remapped","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3170594816,"bytes_avail":286739697664,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":36,"modified":"2023-10-11T11:19:04.644408+0000","services":{}},"progress_events":{}} 2023-10-11T11:19:49.526 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-10-11T11:19:49.527 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-10-11T11:19:49.846 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T11:19:49.848 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2023-10-11T11:19:49.859 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":1585,"stamp":"2023-10-11T11:19:48.654884+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":11,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":107,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3096284,"kb_used_data":418996,"kb_used_omap":292,"kb_used_meta":2676955,"kb_avail":280019236,"statfs":{"total":289910292480,"available":286739697664,"internally_reserved":0,"allocated":429051904,"data_stored":420453125,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299441,"internal_metadata":2741202511},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002455"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":517,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-10-11T11:19:02.772828+0000","last_change":"2023-10-11T11:15:31.661747+0000","last_active":"2023-10-11T11:19:02.772828+0000","last_peered":"2023-10-11T11:19:02.772828+0000","last_clean":"2023-10-11T11:15:27.638215+0000","last_became_active":"2023-10-11T11:15:31.661747+0000","last_became_peered":"2023-10-11T11:15:31.661747+0000","last_unstale":"2023-10-11T11:19:02.772828+0000","last_undegraded":"2023-10-11T11:19:02.772828+0000","last_fullsized":"2023-10-11T11:15:31.649336+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:15:25.047039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":29,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-10-11T11:19:02.774213+0000","last_change":"2023-10-11T11:15:31.671363+0000","last_active":"2023-10-11T11:19:02.774213+0000","last_peered":"2023-10-11T11:19:02.774213+0000","last_clean":"2023-10-11T10:43:40.751300+0000","last_became_active":"2023-10-11T11:15:31.671363+0000","last_became_peered":"2023-10-11T11:15:31.671363+0000","last_unstale":"2023-10-11T11:19:02.774213+0000","last_undegraded":"2023-10-11T11:19:02.774213+0000","last_fullsized":"2023-10-11T11:15:31.652001+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:01:53.884964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":512,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-10-11T11:19:03.429640+0000","last_change":"2023-10-11T10:43:41.772781+0000","last_active":"2023-10-11T11:19:03.429640+0000","last_peered":"2023-10-11T11:19:03.429640+0000","last_clean":"2023-10-11T11:19:03.429640+0000","last_became_active":"2023-10-11T10:43:41.772443+0000","last_became_peered":"2023-10-11T10:43:41.772443+0000","last_unstale":"2023-10-11T11:19:03.429640+0000","last_undegraded":"2023-10-11T11:19:03.429640+0000","last_fullsized":"2023-10-11T11:19:03.429640+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T14:15:30.547284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":513,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-10-11T11:19:02.772211+0000","last_change":"2023-10-11T10:43:41.772615+0000","last_active":"2023-10-11T11:19:02.772211+0000","last_peered":"2023-10-11T11:19:02.772211+0000","last_clean":"2023-10-11T11:19:02.772211+0000","last_became_active":"2023-10-11T10:43:41.772290+0000","last_became_peered":"2023-10-11T10:43:41.772290+0000","last_unstale":"2023-10-11T11:19:02.772211+0000","last_undegraded":"2023-10-11T11:19:02.772211+0000","last_fullsized":"2023-10-11T11:19:02.772211+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:26:06.909963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":648,"reported_epoch":500,"state":"active+undersized+degraded","last_fresh":"2023-10-11T11:19:02.769176+0000","last_change":"2023-10-11T11:15:31.654022+0000","last_active":"2023-10-11T11:19:02.769176+0000","last_peered":"2023-10-11T11:19:02.769176+0000","last_clean":"2023-10-11T11:15:27.636884+0000","last_became_active":"2023-10-11T11:15:31.654022+0000","last_became_peered":"2023-10-11T11:15:31.654022+0000","last_unstale":"2023-10-11T11:19:02.769176+0000","last_undegraded":"2023-10-11T11:15:31.646946+0000","last_fullsized":"2023-10-11T11:15:31.646751+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:36.739637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:36.739637+0000","last_clean_scrub_stamp":"2023-10-11T10:43:36.739637+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T11:33:33.696447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":519,"reported_epoch":499,"state":"active+clean+remapped","last_fresh":"2023-10-11T11:19:01.882635+0000","last_change":"2023-10-11T11:15:33.660200+0000","last_active":"2023-10-11T11:19:01.882635+0000","last_peered":"2023-10-11T11:19:01.882635+0000","last_clean":"2023-10-11T11:19:01.882635+0000","last_became_active":"2023-10-11T11:15:33.659619+0000","last_became_peered":"2023-10-11T11:15:33.659619+0000","last_unstale":"2023-10-11T11:19:01.882635+0000","last_undegraded":"2023-10-11T11:19:01.882635+0000","last_fullsized":"2023-10-11T11:19:01.882635+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T17:21:16.866392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":517,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-10-11T11:19:02.775128+0000","last_change":"2023-10-11T11:15:31.670813+0000","last_active":"2023-10-11T11:19:02.775128+0000","last_peered":"2023-10-11T11:19:02.775128+0000","last_clean":"2023-10-11T11:15:27.638219+0000","last_became_active":"2023-10-11T11:15:31.670813+0000","last_became_peered":"2023-10-11T11:15:31.670813+0000","last_unstale":"2023-10-11T11:19:02.775128+0000","last_undegraded":"2023-10-11T11:19:02.775128+0000","last_fullsized":"2023-10-11T11:15:31.658785+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:41:50.872346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":516,"reported_epoch":499,"state":"active+undersized","last_fresh":"2023-10-11T11:19:00.631668+0000","last_change":"2023-10-11T11:15:31.666378+0000","last_active":"2023-10-11T11:19:00.631668+0000","last_peered":"2023-10-11T11:19:00.631668+0000","last_clean":"2023-10-11T11:15:27.638303+0000","last_became_active":"2023-10-11T11:15:31.666378+0000","last_became_peered":"2023-10-11T11:15:31.666378+0000","last_unstale":"2023-10-11T11:19:00.631668+0000","last_undegraded":"2023-10-11T11:19:00.631668+0000","last_fullsized":"2023-10-11T11:15:31.649186+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T15:35:19.331209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":516,"reported_epoch":499,"state":"active+undersized","last_fresh":"2023-10-11T11:19:00.636960+0000","last_change":"2023-10-11T11:15:31.679465+0000","last_active":"2023-10-11T11:19:00.636960+0000","last_peered":"2023-10-11T11:19:00.636960+0000","last_clean":"2023-10-11T11:15:27.638749+0000","last_became_active":"2023-10-11T11:15:31.679465+0000","last_became_peered":"2023-10-11T11:15:31.679465+0000","last_unstale":"2023-10-11T11:19:00.636960+0000","last_undegraded":"2023-10-11T11:19:00.636960+0000","last_fullsized":"2023-10-11T11:15:31.659135+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T16:03:42.656692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":11,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738805,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":785348,"kb_used_data":5812,"kb_used_omap":96,"kb_used_meta":779423,"kb_avail":93586492,"statfs":{"total":96636764160,"available":95832567808,"internally_reserved":0,"allocated":5951488,"data_stored":3106918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":98609,"internal_metadata":798129871},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 11:14:37 2023","interfaces":[{"interface":"back","average":{"1min":0.314,"5min":0.316,"15min":0.443},"min":{"1min":0.24399999999999999,"5min":0.14799999999999999,"15min":0.14599999999999999},"max":{"1min":0.48499999999999999,"5min":1.712,"15min":9.6400000000000006},"last":0.31},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.35499999999999998,"15min":0.41299999999999998},"min":{"1min":0.20899999999999999,"5min":0.17199999999999999,"15min":0.16400000000000001},"max":{"1min":3.8180000000000001,"5min":3.8180000000000001,"15min":6.3200000000000003},"last":0.26500000000000001}]},{"osd":1,"last update":"Wed Oct 11 11:19:40 2023","interfaces":[{"interface":"back","average":{"1min":0.29599999999999999,"5min":0.28499999999999998,"15min":0.32300000000000001},"min":{"1min":0.17699999999999999,"5min":0.154,"15min":0.154},"max":{"1min":0.40600000000000003,"5min":0.41999999999999998,"15min":2.9540000000000002},"last":0.307},{"interface":"front","average":{"1min":0.317,"5min":0.31,"15min":0.36199999999999999},"min":{"1min":0.20899999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.45000000000000001,"5min":0.49099999999999999,"15min":3.2719999999999998},"last":0.35199999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738756,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1484496,"kb_used_data":406792,"kb_used_omap":107,"kb_used_meta":1077588,"kb_avail":92887344,"statfs":{"total":96636764160,"available":95116640256,"internally_reserved":0,"allocated":416555008,"data_stored":413648921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110309,"internal_metadata":1103450395},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Oct 11 11:14:30 2023","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.35399999999999998,"15min":0.40600000000000003},"min":{"1min":0.18099999999999999,"5min":0.155,"15min":0.14899999999999999},"max":{"1min":1.4319999999999999,"5min":2.3460000000000001,"15min":6.593},"last":0.26200000000000001},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.39100000000000001,"15min":0.48199999999999998},"min":{"1min":0.189,"5min":0.189,"15min":0.183},"max":{"1min":2.2120000000000002,"5min":2.4089999999999998,"15min":6.5960000000000001},"last":0.316}]},{"osd":2,"last update":"Wed Oct 11 11:14:30 2023","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.35499999999999998,"15min":0.47099999999999997},"min":{"1min":0.20300000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":1.2490000000000001,"5min":2.665,"15min":8.4730000000000008},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.375,"15min":0.442},"min":{"1min":0.23200000000000001,"5min":0.20499999999999999,"15min":0.19600000000000001},"max":{"1min":1.28,"5min":2.6230000000000002,"15min":7.0129999999999999},"last":0.33600000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738806,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":826440,"kb_used_data":6392,"kb_used_omap":88,"kb_used_meta":819943,"kb_avail":93545400,"statfs":{"total":96636764160,"available":95790489600,"internally_reserved":0,"allocated":6545408,"data_stored":3697286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":90523,"internal_metadata":839622245},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 11:14:33 2023","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.42799999999999999,"15min":0.41099999999999998},"min":{"1min":0.151,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":1.2809999999999999,"5min":3.5739999999999998,"15min":8.2479999999999993},"last":0.223},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.48299999999999998,"15min":0.49199999999999999},"min":{"1min":0.20699999999999999,"5min":0.182,"15min":0.17999999999999999},"max":{"1min":1.323,"5min":3.6080000000000001,"15min":8.2870000000000008},"last":0.28399999999999997}]},{"osd":2,"last update":"Wed Oct 11 11:19:41 2023","interfaces":[{"interface":"back","average":{"1min":0.314,"5min":0.35399999999999998,"15min":0.35899999999999999},"min":{"1min":0.186,"5min":0.14699999999999999,"15min":0.14699999999999999},"max":{"1min":0.45500000000000002,"5min":7.4829999999999997,"15min":10.617000000000001},"last":0.27600000000000002},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.39400000000000002,"15min":0.35099999999999998},"min":{"1min":0.21199999999999999,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.49099999999999999,"5min":7.5350000000000001,"15min":7.5350000000000001},"last":0.30499999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-11T11:19:51.542 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~259s 2023-10-11T11:19:52.862 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-10-11T11:19:53.194 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T11:19:53.196 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2023-10-11T11:19:53.207 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":1587,"stamp":"2023-10-11T11:19:52.655840+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":11,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":107,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3096284,"kb_used_data":418996,"kb_used_omap":292,"kb_used_meta":2676955,"kb_avail":280019236,"statfs":{"total":289910292480,"available":286739697664,"internally_reserved":0,"allocated":429051904,"data_stored":420453125,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299441,"internal_metadata":2741202511},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002488"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":517,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-10-11T11:19:02.772828+0000","last_change":"2023-10-11T11:15:31.661747+0000","last_active":"2023-10-11T11:19:02.772828+0000","last_peered":"2023-10-11T11:19:02.772828+0000","last_clean":"2023-10-11T11:15:27.638215+0000","last_became_active":"2023-10-11T11:15:31.661747+0000","last_became_peered":"2023-10-11T11:15:31.661747+0000","last_unstale":"2023-10-11T11:19:02.772828+0000","last_undegraded":"2023-10-11T11:19:02.772828+0000","last_fullsized":"2023-10-11T11:15:31.649336+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:15:25.047039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":29,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-10-11T11:19:02.774213+0000","last_change":"2023-10-11T11:15:31.671363+0000","last_active":"2023-10-11T11:19:02.774213+0000","last_peered":"2023-10-11T11:19:02.774213+0000","last_clean":"2023-10-11T10:43:40.751300+0000","last_became_active":"2023-10-11T11:15:31.671363+0000","last_became_peered":"2023-10-11T11:15:31.671363+0000","last_unstale":"2023-10-11T11:19:02.774213+0000","last_undegraded":"2023-10-11T11:19:02.774213+0000","last_fullsized":"2023-10-11T11:15:31.652001+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:01:53.884964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":512,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-10-11T11:19:03.429640+0000","last_change":"2023-10-11T10:43:41.772781+0000","last_active":"2023-10-11T11:19:03.429640+0000","last_peered":"2023-10-11T11:19:03.429640+0000","last_clean":"2023-10-11T11:19:03.429640+0000","last_became_active":"2023-10-11T10:43:41.772443+0000","last_became_peered":"2023-10-11T10:43:41.772443+0000","last_unstale":"2023-10-11T11:19:03.429640+0000","last_undegraded":"2023-10-11T11:19:03.429640+0000","last_fullsized":"2023-10-11T11:19:03.429640+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T14:15:30.547284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":513,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-10-11T11:19:02.772211+0000","last_change":"2023-10-11T10:43:41.772615+0000","last_active":"2023-10-11T11:19:02.772211+0000","last_peered":"2023-10-11T11:19:02.772211+0000","last_clean":"2023-10-11T11:19:02.772211+0000","last_became_active":"2023-10-11T10:43:41.772290+0000","last_became_peered":"2023-10-11T10:43:41.772290+0000","last_unstale":"2023-10-11T11:19:02.772211+0000","last_undegraded":"2023-10-11T11:19:02.772211+0000","last_fullsized":"2023-10-11T11:19:02.772211+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:26:06.909963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":648,"reported_epoch":500,"state":"active+undersized+degraded","last_fresh":"2023-10-11T11:19:02.769176+0000","last_change":"2023-10-11T11:15:31.654022+0000","last_active":"2023-10-11T11:19:02.769176+0000","last_peered":"2023-10-11T11:19:02.769176+0000","last_clean":"2023-10-11T11:15:27.636884+0000","last_became_active":"2023-10-11T11:15:31.654022+0000","last_became_peered":"2023-10-11T11:15:31.654022+0000","last_unstale":"2023-10-11T11:19:02.769176+0000","last_undegraded":"2023-10-11T11:15:31.646946+0000","last_fullsized":"2023-10-11T11:15:31.646751+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:36.739637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:36.739637+0000","last_clean_scrub_stamp":"2023-10-11T10:43:36.739637+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T11:33:33.696447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":520,"reported_epoch":500,"state":"active+clean+remapped","last_fresh":"2023-10-11T11:19:03.429429+0000","last_change":"2023-10-11T11:15:33.660200+0000","last_active":"2023-10-11T11:19:03.429429+0000","last_peered":"2023-10-11T11:19:03.429429+0000","last_clean":"2023-10-11T11:19:03.429429+0000","last_became_active":"2023-10-11T11:15:33.659619+0000","last_became_peered":"2023-10-11T11:15:33.659619+0000","last_unstale":"2023-10-11T11:19:03.429429+0000","last_undegraded":"2023-10-11T11:19:03.429429+0000","last_fullsized":"2023-10-11T11:19:03.429429+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T17:21:16.866392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":517,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-10-11T11:19:02.775128+0000","last_change":"2023-10-11T11:15:31.670813+0000","last_active":"2023-10-11T11:19:02.775128+0000","last_peered":"2023-10-11T11:19:02.775128+0000","last_clean":"2023-10-11T11:15:27.638219+0000","last_became_active":"2023-10-11T11:15:31.670813+0000","last_became_peered":"2023-10-11T11:15:31.670813+0000","last_unstale":"2023-10-11T11:19:02.775128+0000","last_undegraded":"2023-10-11T11:19:02.775128+0000","last_fullsized":"2023-10-11T11:15:31.658785+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:41:50.872346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":516,"reported_epoch":499,"state":"active+undersized","last_fresh":"2023-10-11T11:19:00.631668+0000","last_change":"2023-10-11T11:15:31.666378+0000","last_active":"2023-10-11T11:19:00.631668+0000","last_peered":"2023-10-11T11:19:00.631668+0000","last_clean":"2023-10-11T11:15:27.638303+0000","last_became_active":"2023-10-11T11:15:31.666378+0000","last_became_peered":"2023-10-11T11:15:31.666378+0000","last_unstale":"2023-10-11T11:19:00.631668+0000","last_undegraded":"2023-10-11T11:19:00.631668+0000","last_fullsized":"2023-10-11T11:15:31.649186+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T15:35:19.331209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":516,"reported_epoch":499,"state":"active+undersized","last_fresh":"2023-10-11T11:19:00.636960+0000","last_change":"2023-10-11T11:15:31.679465+0000","last_active":"2023-10-11T11:19:00.636960+0000","last_peered":"2023-10-11T11:19:00.636960+0000","last_clean":"2023-10-11T11:15:27.638749+0000","last_became_active":"2023-10-11T11:15:31.679465+0000","last_became_peered":"2023-10-11T11:15:31.679465+0000","last_unstale":"2023-10-11T11:19:00.636960+0000","last_undegraded":"2023-10-11T11:19:00.636960+0000","last_fullsized":"2023-10-11T11:15:31.659135+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T16:03:42.656692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":11,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738806,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":785348,"kb_used_data":5812,"kb_used_omap":96,"kb_used_meta":779423,"kb_avail":93586492,"statfs":{"total":96636764160,"available":95832567808,"internally_reserved":0,"allocated":5951488,"data_stored":3106918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":98609,"internal_metadata":798129871},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 11:14:37 2023","interfaces":[{"interface":"back","average":{"1min":0.314,"5min":0.316,"15min":0.443},"min":{"1min":0.24399999999999999,"5min":0.14799999999999999,"15min":0.14599999999999999},"max":{"1min":0.48499999999999999,"5min":1.712,"15min":9.6400000000000006},"last":0.31},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.35499999999999998,"15min":0.41299999999999998},"min":{"1min":0.20899999999999999,"5min":0.17199999999999999,"15min":0.16400000000000001},"max":{"1min":3.8180000000000001,"5min":3.8180000000000001,"15min":6.3200000000000003},"last":0.26500000000000001}]},{"osd":1,"last update":"Wed Oct 11 11:19:40 2023","interfaces":[{"interface":"back","average":{"1min":0.29599999999999999,"5min":0.28499999999999998,"15min":0.32300000000000001},"min":{"1min":0.17699999999999999,"5min":0.154,"15min":0.154},"max":{"1min":0.40600000000000003,"5min":0.41999999999999998,"15min":2.9540000000000002},"last":0.24199999999999999},{"interface":"front","average":{"1min":0.317,"5min":0.31,"15min":0.36199999999999999},"min":{"1min":0.20899999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.45000000000000001,"5min":0.49099999999999999,"15min":3.2719999999999998},"last":0.27000000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738756,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1484496,"kb_used_data":406792,"kb_used_omap":107,"kb_used_meta":1077588,"kb_avail":92887344,"statfs":{"total":96636764160,"available":95116640256,"internally_reserved":0,"allocated":416555008,"data_stored":413648921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110309,"internal_metadata":1103450395},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Oct 11 11:14:30 2023","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.35399999999999998,"15min":0.40600000000000003},"min":{"1min":0.18099999999999999,"5min":0.155,"15min":0.14899999999999999},"max":{"1min":1.4319999999999999,"5min":2.3460000000000001,"15min":6.593},"last":0.26200000000000001},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.39100000000000001,"15min":0.48199999999999998},"min":{"1min":0.189,"5min":0.189,"15min":0.183},"max":{"1min":2.2120000000000002,"5min":2.4089999999999998,"15min":6.5960000000000001},"last":0.316}]},{"osd":2,"last update":"Wed Oct 11 11:14:30 2023","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.35499999999999998,"15min":0.47099999999999997},"min":{"1min":0.20300000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":1.2490000000000001,"5min":2.665,"15min":8.4730000000000008},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.375,"15min":0.442},"min":{"1min":0.23200000000000001,"5min":0.20499999999999999,"15min":0.19600000000000001},"max":{"1min":1.28,"5min":2.6230000000000002,"15min":7.0129999999999999},"last":0.33600000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738807,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":826440,"kb_used_data":6392,"kb_used_omap":88,"kb_used_meta":819943,"kb_avail":93545400,"statfs":{"total":96636764160,"available":95790489600,"internally_reserved":0,"allocated":6545408,"data_stored":3697286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":90523,"internal_metadata":839622245},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 11:14:33 2023","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.42799999999999999,"15min":0.41099999999999998},"min":{"1min":0.151,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":1.2809999999999999,"5min":3.5739999999999998,"15min":8.2479999999999993},"last":0.223},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.48299999999999998,"15min":0.49199999999999999},"min":{"1min":0.20699999999999999,"5min":0.182,"15min":0.17999999999999999},"max":{"1min":1.323,"5min":3.6080000000000001,"15min":8.2870000000000008},"last":0.28399999999999997}]},{"osd":2,"last update":"Wed Oct 11 11:19:41 2023","interfaces":[{"interface":"back","average":{"1min":0.314,"5min":0.35399999999999998,"15min":0.35899999999999999},"min":{"1min":0.186,"5min":0.14699999999999999,"15min":0.14699999999999999},"max":{"1min":0.45500000000000002,"5min":7.4829999999999997,"15min":10.617000000000001},"last":0.221},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.39400000000000002,"15min":0.35099999999999998},"min":{"1min":0.21199999999999999,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.49099999999999999,"5min":7.5350000000000001,"15min":7.5350000000000001},"last":0.247}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-11T11:19:53.209 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-10-11T11:19:53.524 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T11:19:53.526 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2023-10-11T11:19:53.538 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":1587,"stamp":"2023-10-11T11:19:52.655840+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":11,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":107,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3096284,"kb_used_data":418996,"kb_used_omap":292,"kb_used_meta":2676955,"kb_avail":280019236,"statfs":{"total":289910292480,"available":286739697664,"internally_reserved":0,"allocated":429051904,"data_stored":420453125,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299441,"internal_metadata":2741202511},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002488"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":517,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-10-11T11:19:02.772828+0000","last_change":"2023-10-11T11:15:31.661747+0000","last_active":"2023-10-11T11:19:02.772828+0000","last_peered":"2023-10-11T11:19:02.772828+0000","last_clean":"2023-10-11T11:15:27.638215+0000","last_became_active":"2023-10-11T11:15:31.661747+0000","last_became_peered":"2023-10-11T11:15:31.661747+0000","last_unstale":"2023-10-11T11:19:02.772828+0000","last_undegraded":"2023-10-11T11:19:02.772828+0000","last_fullsized":"2023-10-11T11:15:31.649336+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:15:25.047039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":29,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-10-11T11:19:02.774213+0000","last_change":"2023-10-11T11:15:31.671363+0000","last_active":"2023-10-11T11:19:02.774213+0000","last_peered":"2023-10-11T11:19:02.774213+0000","last_clean":"2023-10-11T10:43:40.751300+0000","last_became_active":"2023-10-11T11:15:31.671363+0000","last_became_peered":"2023-10-11T11:15:31.671363+0000","last_unstale":"2023-10-11T11:19:02.774213+0000","last_undegraded":"2023-10-11T11:19:02.774213+0000","last_fullsized":"2023-10-11T11:15:31.652001+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:01:53.884964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":512,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-10-11T11:19:03.429640+0000","last_change":"2023-10-11T10:43:41.772781+0000","last_active":"2023-10-11T11:19:03.429640+0000","last_peered":"2023-10-11T11:19:03.429640+0000","last_clean":"2023-10-11T11:19:03.429640+0000","last_became_active":"2023-10-11T10:43:41.772443+0000","last_became_peered":"2023-10-11T10:43:41.772443+0000","last_unstale":"2023-10-11T11:19:03.429640+0000","last_undegraded":"2023-10-11T11:19:03.429640+0000","last_fullsized":"2023-10-11T11:19:03.429640+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T14:15:30.547284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":513,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-10-11T11:19:02.772211+0000","last_change":"2023-10-11T10:43:41.772615+0000","last_active":"2023-10-11T11:19:02.772211+0000","last_peered":"2023-10-11T11:19:02.772211+0000","last_clean":"2023-10-11T11:19:02.772211+0000","last_became_active":"2023-10-11T10:43:41.772290+0000","last_became_peered":"2023-10-11T10:43:41.772290+0000","last_unstale":"2023-10-11T11:19:02.772211+0000","last_undegraded":"2023-10-11T11:19:02.772211+0000","last_fullsized":"2023-10-11T11:19:02.772211+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:26:06.909963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":648,"reported_epoch":500,"state":"active+undersized+degraded","last_fresh":"2023-10-11T11:19:02.769176+0000","last_change":"2023-10-11T11:15:31.654022+0000","last_active":"2023-10-11T11:19:02.769176+0000","last_peered":"2023-10-11T11:19:02.769176+0000","last_clean":"2023-10-11T11:15:27.636884+0000","last_became_active":"2023-10-11T11:15:31.654022+0000","last_became_peered":"2023-10-11T11:15:31.654022+0000","last_unstale":"2023-10-11T11:19:02.769176+0000","last_undegraded":"2023-10-11T11:15:31.646946+0000","last_fullsized":"2023-10-11T11:15:31.646751+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:36.739637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:36.739637+0000","last_clean_scrub_stamp":"2023-10-11T10:43:36.739637+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T11:33:33.696447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":520,"reported_epoch":500,"state":"active+clean+remapped","last_fresh":"2023-10-11T11:19:03.429429+0000","last_change":"2023-10-11T11:15:33.660200+0000","last_active":"2023-10-11T11:19:03.429429+0000","last_peered":"2023-10-11T11:19:03.429429+0000","last_clean":"2023-10-11T11:19:03.429429+0000","last_became_active":"2023-10-11T11:15:33.659619+0000","last_became_peered":"2023-10-11T11:15:33.659619+0000","last_unstale":"2023-10-11T11:19:03.429429+0000","last_undegraded":"2023-10-11T11:19:03.429429+0000","last_fullsized":"2023-10-11T11:19:03.429429+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T17:21:16.866392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":517,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-10-11T11:19:02.775128+0000","last_change":"2023-10-11T11:15:31.670813+0000","last_active":"2023-10-11T11:19:02.775128+0000","last_peered":"2023-10-11T11:19:02.775128+0000","last_clean":"2023-10-11T11:15:27.638219+0000","last_became_active":"2023-10-11T11:15:31.670813+0000","last_became_peered":"2023-10-11T11:15:31.670813+0000","last_unstale":"2023-10-11T11:19:02.775128+0000","last_undegraded":"2023-10-11T11:19:02.775128+0000","last_fullsized":"2023-10-11T11:15:31.658785+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:41:50.872346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":516,"reported_epoch":499,"state":"active+undersized","last_fresh":"2023-10-11T11:19:00.631668+0000","last_change":"2023-10-11T11:15:31.666378+0000","last_active":"2023-10-11T11:19:00.631668+0000","last_peered":"2023-10-11T11:19:00.631668+0000","last_clean":"2023-10-11T11:15:27.638303+0000","last_became_active":"2023-10-11T11:15:31.666378+0000","last_became_peered":"2023-10-11T11:15:31.666378+0000","last_unstale":"2023-10-11T11:19:00.631668+0000","last_undegraded":"2023-10-11T11:19:00.631668+0000","last_fullsized":"2023-10-11T11:15:31.649186+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T15:35:19.331209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":516,"reported_epoch":499,"state":"active+undersized","last_fresh":"2023-10-11T11:19:00.636960+0000","last_change":"2023-10-11T11:15:31.679465+0000","last_active":"2023-10-11T11:19:00.636960+0000","last_peered":"2023-10-11T11:19:00.636960+0000","last_clean":"2023-10-11T11:15:27.638749+0000","last_became_active":"2023-10-11T11:15:31.679465+0000","last_became_peered":"2023-10-11T11:15:31.679465+0000","last_unstale":"2023-10-11T11:19:00.636960+0000","last_undegraded":"2023-10-11T11:19:00.636960+0000","last_fullsized":"2023-10-11T11:15:31.659135+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T16:03:42.656692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":11,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738806,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":785348,"kb_used_data":5812,"kb_used_omap":96,"kb_used_meta":779423,"kb_avail":93586492,"statfs":{"total":96636764160,"available":95832567808,"internally_reserved":0,"allocated":5951488,"data_stored":3106918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":98609,"internal_metadata":798129871},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 11:14:37 2023","interfaces":[{"interface":"back","average":{"1min":0.314,"5min":0.316,"15min":0.443},"min":{"1min":0.24399999999999999,"5min":0.14799999999999999,"15min":0.14599999999999999},"max":{"1min":0.48499999999999999,"5min":1.712,"15min":9.6400000000000006},"last":0.31},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.35499999999999998,"15min":0.41299999999999998},"min":{"1min":0.20899999999999999,"5min":0.17199999999999999,"15min":0.16400000000000001},"max":{"1min":3.8180000000000001,"5min":3.8180000000000001,"15min":6.3200000000000003},"last":0.26500000000000001}]},{"osd":1,"last update":"Wed Oct 11 11:19:40 2023","interfaces":[{"interface":"back","average":{"1min":0.29599999999999999,"5min":0.28499999999999998,"15min":0.32300000000000001},"min":{"1min":0.17699999999999999,"5min":0.154,"15min":0.154},"max":{"1min":0.40600000000000003,"5min":0.41999999999999998,"15min":2.9540000000000002},"last":0.24199999999999999},{"interface":"front","average":{"1min":0.317,"5min":0.31,"15min":0.36199999999999999},"min":{"1min":0.20899999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.45000000000000001,"5min":0.49099999999999999,"15min":3.2719999999999998},"last":0.27000000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738756,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1484496,"kb_used_data":406792,"kb_used_omap":107,"kb_used_meta":1077588,"kb_avail":92887344,"statfs":{"total":96636764160,"available":95116640256,"internally_reserved":0,"allocated":416555008,"data_stored":413648921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110309,"internal_metadata":1103450395},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Oct 11 11:14:30 2023","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.35399999999999998,"15min":0.40600000000000003},"min":{"1min":0.18099999999999999,"5min":0.155,"15min":0.14899999999999999},"max":{"1min":1.4319999999999999,"5min":2.3460000000000001,"15min":6.593},"last":0.26200000000000001},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.39100000000000001,"15min":0.48199999999999998},"min":{"1min":0.189,"5min":0.189,"15min":0.183},"max":{"1min":2.2120000000000002,"5min":2.4089999999999998,"15min":6.5960000000000001},"last":0.316}]},{"osd":2,"last update":"Wed Oct 11 11:14:30 2023","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.35499999999999998,"15min":0.47099999999999997},"min":{"1min":0.20300000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":1.2490000000000001,"5min":2.665,"15min":8.4730000000000008},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.375,"15min":0.442},"min":{"1min":0.23200000000000001,"5min":0.20499999999999999,"15min":0.19600000000000001},"max":{"1min":1.28,"5min":2.6230000000000002,"15min":7.0129999999999999},"last":0.33600000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738807,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":826440,"kb_used_data":6392,"kb_used_omap":88,"kb_used_meta":819943,"kb_avail":93545400,"statfs":{"total":96636764160,"available":95790489600,"internally_reserved":0,"allocated":6545408,"data_stored":3697286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":90523,"internal_metadata":839622245},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 11:14:33 2023","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.42799999999999999,"15min":0.41099999999999998},"min":{"1min":0.151,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":1.2809999999999999,"5min":3.5739999999999998,"15min":8.2479999999999993},"last":0.223},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.48299999999999998,"15min":0.49199999999999999},"min":{"1min":0.20699999999999999,"5min":0.182,"15min":0.17999999999999999},"max":{"1min":1.323,"5min":3.6080000000000001,"15min":8.2870000000000008},"last":0.28399999999999997}]},{"osd":2,"last update":"Wed Oct 11 11:19:41 2023","interfaces":[{"interface":"back","average":{"1min":0.314,"5min":0.35399999999999998,"15min":0.35899999999999999},"min":{"1min":0.186,"5min":0.14699999999999999,"15min":0.14699999999999999},"max":{"1min":0.45500000000000002,"5min":7.4829999999999997,"15min":10.617000000000001},"last":0.221},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.39400000000000002,"15min":0.35099999999999998},"min":{"1min":0.21199999999999999,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.49099999999999999,"5min":7.5350000000000001,"15min":7.5350000000000001},"last":0.247}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-11T11:19:53.539 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-10-11T11:19:53.539 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 517, 'reported_epoch': 500, 'state': 'active+undersized', 'last_fresh': '2023-10-11T11:19:02.772828+0000', 'last_change': '2023-10-11T11:15:31.661747+0000', 'last_active': '2023-10-11T11:19:02.772828+0000', 'last_peered': '2023-10-11T11:19:02.772828+0000', 'last_clean': '2023-10-11T11:15:27.638215+0000', 'last_became_active': '2023-10-11T11:15:31.661747+0000', 'last_became_peered': '2023-10-11T11:15:31.661747+0000', 'last_unstale': '2023-10-11T11:19:02.772828+0000', 'last_undegraded': '2023-10-11T11:19:02.772828+0000', 'last_fullsized': '2023-10-11T11:15:31.649336+0000', 'mapping_epoch': 480, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_clean_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-12T22:15:25.047039+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-10-11T11:19:53.540 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-10-11T11:19:53.540 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 29, 'reported_epoch': 500, 'state': 'active+undersized', 'last_fresh': '2023-10-11T11:19:02.774213+0000', 'last_change': '2023-10-11T11:15:31.671363+0000', 'last_active': '2023-10-11T11:19:02.774213+0000', 'last_peered': '2023-10-11T11:19:02.774213+0000', 'last_clean': '2023-10-11T10:43:40.751300+0000', 'last_became_active': '2023-10-11T11:15:31.671363+0000', 'last_became_peered': '2023-10-11T11:15:31.671363+0000', 'last_unstale': '2023-10-11T11:19:02.774213+0000', 'last_undegraded': '2023-10-11T11:19:02.774213+0000', 'last_fullsized': '2023-10-11T11:15:31.652001+0000', 'mapping_epoch': 480, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_clean_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-12T21:01:53.884964+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-10-11T11:19:53.540 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-10-11T11:19:53.541 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "11'97", 'reported_seq': 648, 'reported_epoch': 500, 'state': 'active+undersized+degraded', 'last_fresh': '2023-10-11T11:19:02.769176+0000', 'last_change': '2023-10-11T11:15:31.654022+0000', 'last_active': '2023-10-11T11:19:02.769176+0000', 'last_peered': '2023-10-11T11:19:02.769176+0000', 'last_clean': '2023-10-11T11:15:27.636884+0000', 'last_became_active': '2023-10-11T11:15:31.654022+0000', 'last_became_peered': '2023-10-11T11:15:31.654022+0000', 'last_unstale': '2023-10-11T11:19:02.769176+0000', 'last_undegraded': '2023-10-11T11:15:31.646946+0000', 'last_fullsized': '2023-10-11T11:15:31.646751+0000', 'mapping_epoch': 480, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-11T10:43:36.739637+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-11T10:43:36.739637+0000', 'last_clean_scrub_stamp': '2023-10-11T10:43:36.739637+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-12T11:33:33.696447+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-10-11T11:19:53.541 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.0 is not active+clean 2023-10-11T11:19:53.541 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 520, 'reported_epoch': 500, 'state': 'active+clean+remapped', 'last_fresh': '2023-10-11T11:19:03.429429+0000', 'last_change': '2023-10-11T11:15:33.660200+0000', 'last_active': '2023-10-11T11:19:03.429429+0000', 'last_peered': '2023-10-11T11:19:03.429429+0000', 'last_clean': '2023-10-11T11:19:03.429429+0000', 'last_became_active': '2023-10-11T11:15:33.659619+0000', 'last_became_peered': '2023-10-11T11:15:33.659619+0000', 'last_unstale': '2023-10-11T11:19:03.429429+0000', 'last_undegraded': '2023-10-11T11:19:03.429429+0000', 'last_fullsized': '2023-10-11T11:19:03.429429+0000', 'mapping_epoch': 482, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 483, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_clean_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-12T17:21:16.866392+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2, 1], 'avail_no_missing': ['2', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-10-11T11:19:53.541 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-10-11T11:19:53.542 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 517, 'reported_epoch': 500, 'state': 'active+undersized', 'last_fresh': '2023-10-11T11:19:02.775128+0000', 'last_change': '2023-10-11T11:15:31.670813+0000', 'last_active': '2023-10-11T11:19:02.775128+0000', 'last_peered': '2023-10-11T11:19:02.775128+0000', 'last_clean': '2023-10-11T11:15:27.638219+0000', 'last_became_active': '2023-10-11T11:15:31.670813+0000', 'last_became_peered': '2023-10-11T11:15:31.670813+0000', 'last_unstale': '2023-10-11T11:19:02.775128+0000', 'last_undegraded': '2023-10-11T11:19:02.775128+0000', 'last_fullsized': '2023-10-11T11:15:31.658785+0000', 'mapping_epoch': 480, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_clean_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-12T21:41:50.872346+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-10-11T11:19:53.542 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-10-11T11:19:53.542 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 516, 'reported_epoch': 499, 'state': 'active+undersized', 'last_fresh': '2023-10-11T11:19:00.631668+0000', 'last_change': '2023-10-11T11:15:31.666378+0000', 'last_active': '2023-10-11T11:19:00.631668+0000', 'last_peered': '2023-10-11T11:19:00.631668+0000', 'last_clean': '2023-10-11T11:15:27.638303+0000', 'last_became_active': '2023-10-11T11:15:31.666378+0000', 'last_became_peered': '2023-10-11T11:15:31.666378+0000', 'last_unstale': '2023-10-11T11:19:00.631668+0000', 'last_undegraded': '2023-10-11T11:19:00.631668+0000', 'last_fullsized': '2023-10-11T11:15:31.649186+0000', 'mapping_epoch': 480, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_clean_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-12T15:35:19.331209+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-10-11T11:19:53.543 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-10-11T11:19:53.543 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 516, 'reported_epoch': 499, 'state': 'active+undersized', 'last_fresh': '2023-10-11T11:19:00.636960+0000', 'last_change': '2023-10-11T11:15:31.679465+0000', 'last_active': '2023-10-11T11:19:00.636960+0000', 'last_peered': '2023-10-11T11:19:00.636960+0000', 'last_clean': '2023-10-11T11:15:27.638749+0000', 'last_became_active': '2023-10-11T11:15:31.679465+0000', 'last_became_peered': '2023-10-11T11:15:31.679465+0000', 'last_unstale': '2023-10-11T11:19:00.636960+0000', 'last_undegraded': '2023-10-11T11:19:00.636960+0000', 'last_fullsized': '2023-10-11T11:15:31.659135+0000', 'mapping_epoch': 480, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_clean_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-12T16:03:42.656692+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-10-11T11:19:53.543 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-10-11T11:19:53.942 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T11:19:53.942 INFO:teuthology.orchestra.run.smithi111.stdout:{"fsid":"022ce6f2-9a5b-45ad-8b77-a1332d92f228","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2189,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":500,"num_osds":3,"num_up_osds":2,"osd_up_since":1697022930,"num_in_osds":3,"osd_in_since":1697021005,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":2},{"state_name":"active+undersized+degraded","count":1},{"state_name":"active+clean+remapped","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3170594816,"bytes_avail":286739697664,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":36,"modified":"2023-10-11T11:19:04.644408+0000","services":{}},"progress_events":{}} 2023-10-11T11:19:53.956 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-10-11T11:19:53.956 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-10-11T11:19:54.272 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T11:19:54.274 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2023-10-11T11:19:54.286 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":1587,"stamp":"2023-10-11T11:19:52.655840+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":11,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":107,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3096284,"kb_used_data":418996,"kb_used_omap":292,"kb_used_meta":2676955,"kb_avail":280019236,"statfs":{"total":289910292480,"available":286739697664,"internally_reserved":0,"allocated":429051904,"data_stored":420453125,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299441,"internal_metadata":2741202511},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002488"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":517,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-10-11T11:19:02.772828+0000","last_change":"2023-10-11T11:15:31.661747+0000","last_active":"2023-10-11T11:19:02.772828+0000","last_peered":"2023-10-11T11:19:02.772828+0000","last_clean":"2023-10-11T11:15:27.638215+0000","last_became_active":"2023-10-11T11:15:31.661747+0000","last_became_peered":"2023-10-11T11:15:31.661747+0000","last_unstale":"2023-10-11T11:19:02.772828+0000","last_undegraded":"2023-10-11T11:19:02.772828+0000","last_fullsized":"2023-10-11T11:15:31.649336+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:15:25.047039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":29,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-10-11T11:19:02.774213+0000","last_change":"2023-10-11T11:15:31.671363+0000","last_active":"2023-10-11T11:19:02.774213+0000","last_peered":"2023-10-11T11:19:02.774213+0000","last_clean":"2023-10-11T10:43:40.751300+0000","last_became_active":"2023-10-11T11:15:31.671363+0000","last_became_peered":"2023-10-11T11:15:31.671363+0000","last_unstale":"2023-10-11T11:19:02.774213+0000","last_undegraded":"2023-10-11T11:19:02.774213+0000","last_fullsized":"2023-10-11T11:15:31.652001+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:01:53.884964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":512,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-10-11T11:19:03.429640+0000","last_change":"2023-10-11T10:43:41.772781+0000","last_active":"2023-10-11T11:19:03.429640+0000","last_peered":"2023-10-11T11:19:03.429640+0000","last_clean":"2023-10-11T11:19:03.429640+0000","last_became_active":"2023-10-11T10:43:41.772443+0000","last_became_peered":"2023-10-11T10:43:41.772443+0000","last_unstale":"2023-10-11T11:19:03.429640+0000","last_undegraded":"2023-10-11T11:19:03.429640+0000","last_fullsized":"2023-10-11T11:19:03.429640+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T14:15:30.547284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":513,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-10-11T11:19:02.772211+0000","last_change":"2023-10-11T10:43:41.772615+0000","last_active":"2023-10-11T11:19:02.772211+0000","last_peered":"2023-10-11T11:19:02.772211+0000","last_clean":"2023-10-11T11:19:02.772211+0000","last_became_active":"2023-10-11T10:43:41.772290+0000","last_became_peered":"2023-10-11T10:43:41.772290+0000","last_unstale":"2023-10-11T11:19:02.772211+0000","last_undegraded":"2023-10-11T11:19:02.772211+0000","last_fullsized":"2023-10-11T11:19:02.772211+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:26:06.909963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":648,"reported_epoch":500,"state":"active+undersized+degraded","last_fresh":"2023-10-11T11:19:02.769176+0000","last_change":"2023-10-11T11:15:31.654022+0000","last_active":"2023-10-11T11:19:02.769176+0000","last_peered":"2023-10-11T11:19:02.769176+0000","last_clean":"2023-10-11T11:15:27.636884+0000","last_became_active":"2023-10-11T11:15:31.654022+0000","last_became_peered":"2023-10-11T11:15:31.654022+0000","last_unstale":"2023-10-11T11:19:02.769176+0000","last_undegraded":"2023-10-11T11:15:31.646946+0000","last_fullsized":"2023-10-11T11:15:31.646751+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:36.739637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:36.739637+0000","last_clean_scrub_stamp":"2023-10-11T10:43:36.739637+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T11:33:33.696447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":520,"reported_epoch":500,"state":"active+clean+remapped","last_fresh":"2023-10-11T11:19:03.429429+0000","last_change":"2023-10-11T11:15:33.660200+0000","last_active":"2023-10-11T11:19:03.429429+0000","last_peered":"2023-10-11T11:19:03.429429+0000","last_clean":"2023-10-11T11:19:03.429429+0000","last_became_active":"2023-10-11T11:15:33.659619+0000","last_became_peered":"2023-10-11T11:15:33.659619+0000","last_unstale":"2023-10-11T11:19:03.429429+0000","last_undegraded":"2023-10-11T11:19:03.429429+0000","last_fullsized":"2023-10-11T11:19:03.429429+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T17:21:16.866392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":517,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-10-11T11:19:02.775128+0000","last_change":"2023-10-11T11:15:31.670813+0000","last_active":"2023-10-11T11:19:02.775128+0000","last_peered":"2023-10-11T11:19:02.775128+0000","last_clean":"2023-10-11T11:15:27.638219+0000","last_became_active":"2023-10-11T11:15:31.670813+0000","last_became_peered":"2023-10-11T11:15:31.670813+0000","last_unstale":"2023-10-11T11:19:02.775128+0000","last_undegraded":"2023-10-11T11:19:02.775128+0000","last_fullsized":"2023-10-11T11:15:31.658785+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:41:50.872346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":516,"reported_epoch":499,"state":"active+undersized","last_fresh":"2023-10-11T11:19:00.631668+0000","last_change":"2023-10-11T11:15:31.666378+0000","last_active":"2023-10-11T11:19:00.631668+0000","last_peered":"2023-10-11T11:19:00.631668+0000","last_clean":"2023-10-11T11:15:27.638303+0000","last_became_active":"2023-10-11T11:15:31.666378+0000","last_became_peered":"2023-10-11T11:15:31.666378+0000","last_unstale":"2023-10-11T11:19:00.631668+0000","last_undegraded":"2023-10-11T11:19:00.631668+0000","last_fullsized":"2023-10-11T11:15:31.649186+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T15:35:19.331209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":516,"reported_epoch":499,"state":"active+undersized","last_fresh":"2023-10-11T11:19:00.636960+0000","last_change":"2023-10-11T11:15:31.679465+0000","last_active":"2023-10-11T11:19:00.636960+0000","last_peered":"2023-10-11T11:19:00.636960+0000","last_clean":"2023-10-11T11:15:27.638749+0000","last_became_active":"2023-10-11T11:15:31.679465+0000","last_became_peered":"2023-10-11T11:15:31.679465+0000","last_unstale":"2023-10-11T11:19:00.636960+0000","last_undegraded":"2023-10-11T11:19:00.636960+0000","last_fullsized":"2023-10-11T11:15:31.659135+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T16:03:42.656692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":11,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738806,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":785348,"kb_used_data":5812,"kb_used_omap":96,"kb_used_meta":779423,"kb_avail":93586492,"statfs":{"total":96636764160,"available":95832567808,"internally_reserved":0,"allocated":5951488,"data_stored":3106918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":98609,"internal_metadata":798129871},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 11:14:37 2023","interfaces":[{"interface":"back","average":{"1min":0.314,"5min":0.316,"15min":0.443},"min":{"1min":0.24399999999999999,"5min":0.14799999999999999,"15min":0.14599999999999999},"max":{"1min":0.48499999999999999,"5min":1.712,"15min":9.6400000000000006},"last":0.31},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.35499999999999998,"15min":0.41299999999999998},"min":{"1min":0.20899999999999999,"5min":0.17199999999999999,"15min":0.16400000000000001},"max":{"1min":3.8180000000000001,"5min":3.8180000000000001,"15min":6.3200000000000003},"last":0.26500000000000001}]},{"osd":1,"last update":"Wed Oct 11 11:19:40 2023","interfaces":[{"interface":"back","average":{"1min":0.29599999999999999,"5min":0.28499999999999998,"15min":0.32300000000000001},"min":{"1min":0.17699999999999999,"5min":0.154,"15min":0.154},"max":{"1min":0.40600000000000003,"5min":0.41999999999999998,"15min":2.9540000000000002},"last":0.24199999999999999},{"interface":"front","average":{"1min":0.317,"5min":0.31,"15min":0.36199999999999999},"min":{"1min":0.20899999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.45000000000000001,"5min":0.49099999999999999,"15min":3.2719999999999998},"last":0.27000000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738756,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1484496,"kb_used_data":406792,"kb_used_omap":107,"kb_used_meta":1077588,"kb_avail":92887344,"statfs":{"total":96636764160,"available":95116640256,"internally_reserved":0,"allocated":416555008,"data_stored":413648921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110309,"internal_metadata":1103450395},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Oct 11 11:14:30 2023","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.35399999999999998,"15min":0.40600000000000003},"min":{"1min":0.18099999999999999,"5min":0.155,"15min":0.14899999999999999},"max":{"1min":1.4319999999999999,"5min":2.3460000000000001,"15min":6.593},"last":0.26200000000000001},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.39100000000000001,"15min":0.48199999999999998},"min":{"1min":0.189,"5min":0.189,"15min":0.183},"max":{"1min":2.2120000000000002,"5min":2.4089999999999998,"15min":6.5960000000000001},"last":0.316}]},{"osd":2,"last update":"Wed Oct 11 11:14:30 2023","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.35499999999999998,"15min":0.47099999999999997},"min":{"1min":0.20300000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":1.2490000000000001,"5min":2.665,"15min":8.4730000000000008},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.375,"15min":0.442},"min":{"1min":0.23200000000000001,"5min":0.20499999999999999,"15min":0.19600000000000001},"max":{"1min":1.28,"5min":2.6230000000000002,"15min":7.0129999999999999},"last":0.33600000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738807,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":826440,"kb_used_data":6392,"kb_used_omap":88,"kb_used_meta":819943,"kb_avail":93545400,"statfs":{"total":96636764160,"available":95790489600,"internally_reserved":0,"allocated":6545408,"data_stored":3697286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":90523,"internal_metadata":839622245},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 11:14:33 2023","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.42799999999999999,"15min":0.41099999999999998},"min":{"1min":0.151,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":1.2809999999999999,"5min":3.5739999999999998,"15min":8.2479999999999993},"last":0.223},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.48299999999999998,"15min":0.49199999999999999},"min":{"1min":0.20699999999999999,"5min":0.182,"15min":0.17999999999999999},"max":{"1min":1.323,"5min":3.6080000000000001,"15min":8.2870000000000008},"last":0.28399999999999997}]},{"osd":2,"last update":"Wed Oct 11 11:19:41 2023","interfaces":[{"interface":"back","average":{"1min":0.314,"5min":0.35399999999999998,"15min":0.35899999999999999},"min":{"1min":0.186,"5min":0.14699999999999999,"15min":0.14699999999999999},"max":{"1min":0.45500000000000002,"5min":7.4829999999999997,"15min":10.617000000000001},"last":0.221},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.39400000000000002,"15min":0.35099999999999998},"min":{"1min":0.21199999999999999,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.49099999999999999,"5min":7.5350000000000001,"15min":7.5350000000000001},"last":0.247}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-11T11:19:56.945 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~265s 2023-10-11T11:19:57.287 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-10-11T11:19:57.620 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T11:19:57.623 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2023-10-11T11:19:57.634 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":1589,"stamp":"2023-10-11T11:19:56.656802+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":11,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":107,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3096284,"kb_used_data":418996,"kb_used_omap":292,"kb_used_meta":2676955,"kb_avail":280019236,"statfs":{"total":289910292480,"available":286739697664,"internally_reserved":0,"allocated":429051904,"data_stored":420453125,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299441,"internal_metadata":2741202511},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002668"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":517,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-10-11T11:19:02.772828+0000","last_change":"2023-10-11T11:15:31.661747+0000","last_active":"2023-10-11T11:19:02.772828+0000","last_peered":"2023-10-11T11:19:02.772828+0000","last_clean":"2023-10-11T11:15:27.638215+0000","last_became_active":"2023-10-11T11:15:31.661747+0000","last_became_peered":"2023-10-11T11:15:31.661747+0000","last_unstale":"2023-10-11T11:19:02.772828+0000","last_undegraded":"2023-10-11T11:19:02.772828+0000","last_fullsized":"2023-10-11T11:15:31.649336+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:15:25.047039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":29,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-10-11T11:19:02.774213+0000","last_change":"2023-10-11T11:15:31.671363+0000","last_active":"2023-10-11T11:19:02.774213+0000","last_peered":"2023-10-11T11:19:02.774213+0000","last_clean":"2023-10-11T10:43:40.751300+0000","last_became_active":"2023-10-11T11:15:31.671363+0000","last_became_peered":"2023-10-11T11:15:31.671363+0000","last_unstale":"2023-10-11T11:19:02.774213+0000","last_undegraded":"2023-10-11T11:19:02.774213+0000","last_fullsized":"2023-10-11T11:15:31.652001+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:01:53.884964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":512,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-10-11T11:19:03.429640+0000","last_change":"2023-10-11T10:43:41.772781+0000","last_active":"2023-10-11T11:19:03.429640+0000","last_peered":"2023-10-11T11:19:03.429640+0000","last_clean":"2023-10-11T11:19:03.429640+0000","last_became_active":"2023-10-11T10:43:41.772443+0000","last_became_peered":"2023-10-11T10:43:41.772443+0000","last_unstale":"2023-10-11T11:19:03.429640+0000","last_undegraded":"2023-10-11T11:19:03.429640+0000","last_fullsized":"2023-10-11T11:19:03.429640+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T14:15:30.547284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":513,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-10-11T11:19:02.772211+0000","last_change":"2023-10-11T10:43:41.772615+0000","last_active":"2023-10-11T11:19:02.772211+0000","last_peered":"2023-10-11T11:19:02.772211+0000","last_clean":"2023-10-11T11:19:02.772211+0000","last_became_active":"2023-10-11T10:43:41.772290+0000","last_became_peered":"2023-10-11T10:43:41.772290+0000","last_unstale":"2023-10-11T11:19:02.772211+0000","last_undegraded":"2023-10-11T11:19:02.772211+0000","last_fullsized":"2023-10-11T11:19:02.772211+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:26:06.909963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":648,"reported_epoch":500,"state":"active+undersized+degraded","last_fresh":"2023-10-11T11:19:02.769176+0000","last_change":"2023-10-11T11:15:31.654022+0000","last_active":"2023-10-11T11:19:02.769176+0000","last_peered":"2023-10-11T11:19:02.769176+0000","last_clean":"2023-10-11T11:15:27.636884+0000","last_became_active":"2023-10-11T11:15:31.654022+0000","last_became_peered":"2023-10-11T11:15:31.654022+0000","last_unstale":"2023-10-11T11:19:02.769176+0000","last_undegraded":"2023-10-11T11:15:31.646946+0000","last_fullsized":"2023-10-11T11:15:31.646751+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:36.739637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:36.739637+0000","last_clean_scrub_stamp":"2023-10-11T10:43:36.739637+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T11:33:33.696447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":520,"reported_epoch":500,"state":"active+clean+remapped","last_fresh":"2023-10-11T11:19:03.429429+0000","last_change":"2023-10-11T11:15:33.660200+0000","last_active":"2023-10-11T11:19:03.429429+0000","last_peered":"2023-10-11T11:19:03.429429+0000","last_clean":"2023-10-11T11:19:03.429429+0000","last_became_active":"2023-10-11T11:15:33.659619+0000","last_became_peered":"2023-10-11T11:15:33.659619+0000","last_unstale":"2023-10-11T11:19:03.429429+0000","last_undegraded":"2023-10-11T11:19:03.429429+0000","last_fullsized":"2023-10-11T11:19:03.429429+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T17:21:16.866392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":517,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-10-11T11:19:02.775128+0000","last_change":"2023-10-11T11:15:31.670813+0000","last_active":"2023-10-11T11:19:02.775128+0000","last_peered":"2023-10-11T11:19:02.775128+0000","last_clean":"2023-10-11T11:15:27.638219+0000","last_became_active":"2023-10-11T11:15:31.670813+0000","last_became_peered":"2023-10-11T11:15:31.670813+0000","last_unstale":"2023-10-11T11:19:02.775128+0000","last_undegraded":"2023-10-11T11:19:02.775128+0000","last_fullsized":"2023-10-11T11:15:31.658785+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:41:50.872346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":516,"reported_epoch":499,"state":"active+undersized","last_fresh":"2023-10-11T11:19:00.631668+0000","last_change":"2023-10-11T11:15:31.666378+0000","last_active":"2023-10-11T11:19:00.631668+0000","last_peered":"2023-10-11T11:19:00.631668+0000","last_clean":"2023-10-11T11:15:27.638303+0000","last_became_active":"2023-10-11T11:15:31.666378+0000","last_became_peered":"2023-10-11T11:15:31.666378+0000","last_unstale":"2023-10-11T11:19:00.631668+0000","last_undegraded":"2023-10-11T11:19:00.631668+0000","last_fullsized":"2023-10-11T11:15:31.649186+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T15:35:19.331209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":516,"reported_epoch":499,"state":"active+undersized","last_fresh":"2023-10-11T11:19:00.636960+0000","last_change":"2023-10-11T11:15:31.679465+0000","last_active":"2023-10-11T11:19:00.636960+0000","last_peered":"2023-10-11T11:19:00.636960+0000","last_clean":"2023-10-11T11:15:27.638749+0000","last_became_active":"2023-10-11T11:15:31.679465+0000","last_became_peered":"2023-10-11T11:15:31.679465+0000","last_unstale":"2023-10-11T11:19:00.636960+0000","last_undegraded":"2023-10-11T11:19:00.636960+0000","last_fullsized":"2023-10-11T11:15:31.659135+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T16:03:42.656692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":11,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738807,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":785348,"kb_used_data":5812,"kb_used_omap":96,"kb_used_meta":779423,"kb_avail":93586492,"statfs":{"total":96636764160,"available":95832567808,"internally_reserved":0,"allocated":5951488,"data_stored":3106918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":98609,"internal_metadata":798129871},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 11:14:37 2023","interfaces":[{"interface":"back","average":{"1min":0.314,"5min":0.316,"15min":0.443},"min":{"1min":0.24399999999999999,"5min":0.14799999999999999,"15min":0.14599999999999999},"max":{"1min":0.48499999999999999,"5min":1.712,"15min":9.6400000000000006},"last":0.31},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.35499999999999998,"15min":0.41299999999999998},"min":{"1min":0.20899999999999999,"5min":0.17199999999999999,"15min":0.16400000000000001},"max":{"1min":3.8180000000000001,"5min":3.8180000000000001,"15min":6.3200000000000003},"last":0.26500000000000001}]},{"osd":1,"last update":"Wed Oct 11 11:19:40 2023","interfaces":[{"interface":"back","average":{"1min":0.29599999999999999,"5min":0.28499999999999998,"15min":0.32300000000000001},"min":{"1min":0.17699999999999999,"5min":0.154,"15min":0.154},"max":{"1min":0.40600000000000003,"5min":0.41999999999999998,"15min":2.9540000000000002},"last":0.27000000000000002},{"interface":"front","average":{"1min":0.317,"5min":0.31,"15min":0.36199999999999999},"min":{"1min":0.20899999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.45000000000000001,"5min":0.49099999999999999,"15min":3.2719999999999998},"last":0.30599999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738756,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1484496,"kb_used_data":406792,"kb_used_omap":107,"kb_used_meta":1077588,"kb_avail":92887344,"statfs":{"total":96636764160,"available":95116640256,"internally_reserved":0,"allocated":416555008,"data_stored":413648921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110309,"internal_metadata":1103450395},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Oct 11 11:14:30 2023","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.35399999999999998,"15min":0.40600000000000003},"min":{"1min":0.18099999999999999,"5min":0.155,"15min":0.14899999999999999},"max":{"1min":1.4319999999999999,"5min":2.3460000000000001,"15min":6.593},"last":0.26200000000000001},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.39100000000000001,"15min":0.48199999999999998},"min":{"1min":0.189,"5min":0.189,"15min":0.183},"max":{"1min":2.2120000000000002,"5min":2.4089999999999998,"15min":6.5960000000000001},"last":0.316}]},{"osd":2,"last update":"Wed Oct 11 11:14:30 2023","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.35499999999999998,"15min":0.47099999999999997},"min":{"1min":0.20300000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":1.2490000000000001,"5min":2.665,"15min":8.4730000000000008},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.375,"15min":0.442},"min":{"1min":0.23200000000000001,"5min":0.20499999999999999,"15min":0.19600000000000001},"max":{"1min":1.28,"5min":2.6230000000000002,"15min":7.0129999999999999},"last":0.33600000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738808,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":826440,"kb_used_data":6392,"kb_used_omap":88,"kb_used_meta":819943,"kb_avail":93545400,"statfs":{"total":96636764160,"available":95790489600,"internally_reserved":0,"allocated":6545408,"data_stored":3697286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":90523,"internal_metadata":839622245},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 11:14:33 2023","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.42799999999999999,"15min":0.41099999999999998},"min":{"1min":0.151,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":1.2809999999999999,"5min":3.5739999999999998,"15min":8.2479999999999993},"last":0.223},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.48299999999999998,"15min":0.49199999999999999},"min":{"1min":0.20699999999999999,"5min":0.182,"15min":0.17999999999999999},"max":{"1min":1.323,"5min":3.6080000000000001,"15min":8.2870000000000008},"last":0.28399999999999997}]},{"osd":2,"last update":"Wed Oct 11 11:19:41 2023","interfaces":[{"interface":"back","average":{"1min":0.314,"5min":0.35399999999999998,"15min":0.35899999999999999},"min":{"1min":0.186,"5min":0.14699999999999999,"15min":0.14699999999999999},"max":{"1min":0.45500000000000002,"5min":7.4829999999999997,"15min":10.617000000000001},"last":0.312},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.39400000000000002,"15min":0.35099999999999998},"min":{"1min":0.21199999999999999,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.49099999999999999,"5min":7.5350000000000001,"15min":7.5350000000000001},"last":0.34000000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-11T11:19:57.635 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-10-11T11:19:57.952 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T11:19:57.954 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2023-10-11T11:19:57.965 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":1589,"stamp":"2023-10-11T11:19:56.656802+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":11,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":107,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3096284,"kb_used_data":418996,"kb_used_omap":292,"kb_used_meta":2676955,"kb_avail":280019236,"statfs":{"total":289910292480,"available":286739697664,"internally_reserved":0,"allocated":429051904,"data_stored":420453125,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299441,"internal_metadata":2741202511},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002668"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":517,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-10-11T11:19:02.772828+0000","last_change":"2023-10-11T11:15:31.661747+0000","last_active":"2023-10-11T11:19:02.772828+0000","last_peered":"2023-10-11T11:19:02.772828+0000","last_clean":"2023-10-11T11:15:27.638215+0000","last_became_active":"2023-10-11T11:15:31.661747+0000","last_became_peered":"2023-10-11T11:15:31.661747+0000","last_unstale":"2023-10-11T11:19:02.772828+0000","last_undegraded":"2023-10-11T11:19:02.772828+0000","last_fullsized":"2023-10-11T11:15:31.649336+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:15:25.047039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":29,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-10-11T11:19:02.774213+0000","last_change":"2023-10-11T11:15:31.671363+0000","last_active":"2023-10-11T11:19:02.774213+0000","last_peered":"2023-10-11T11:19:02.774213+0000","last_clean":"2023-10-11T10:43:40.751300+0000","last_became_active":"2023-10-11T11:15:31.671363+0000","last_became_peered":"2023-10-11T11:15:31.671363+0000","last_unstale":"2023-10-11T11:19:02.774213+0000","last_undegraded":"2023-10-11T11:19:02.774213+0000","last_fullsized":"2023-10-11T11:15:31.652001+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:01:53.884964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":512,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-10-11T11:19:03.429640+0000","last_change":"2023-10-11T10:43:41.772781+0000","last_active":"2023-10-11T11:19:03.429640+0000","last_peered":"2023-10-11T11:19:03.429640+0000","last_clean":"2023-10-11T11:19:03.429640+0000","last_became_active":"2023-10-11T10:43:41.772443+0000","last_became_peered":"2023-10-11T10:43:41.772443+0000","last_unstale":"2023-10-11T11:19:03.429640+0000","last_undegraded":"2023-10-11T11:19:03.429640+0000","last_fullsized":"2023-10-11T11:19:03.429640+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T14:15:30.547284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":513,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-10-11T11:19:02.772211+0000","last_change":"2023-10-11T10:43:41.772615+0000","last_active":"2023-10-11T11:19:02.772211+0000","last_peered":"2023-10-11T11:19:02.772211+0000","last_clean":"2023-10-11T11:19:02.772211+0000","last_became_active":"2023-10-11T10:43:41.772290+0000","last_became_peered":"2023-10-11T10:43:41.772290+0000","last_unstale":"2023-10-11T11:19:02.772211+0000","last_undegraded":"2023-10-11T11:19:02.772211+0000","last_fullsized":"2023-10-11T11:19:02.772211+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:26:06.909963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":648,"reported_epoch":500,"state":"active+undersized+degraded","last_fresh":"2023-10-11T11:19:02.769176+0000","last_change":"2023-10-11T11:15:31.654022+0000","last_active":"2023-10-11T11:19:02.769176+0000","last_peered":"2023-10-11T11:19:02.769176+0000","last_clean":"2023-10-11T11:15:27.636884+0000","last_became_active":"2023-10-11T11:15:31.654022+0000","last_became_peered":"2023-10-11T11:15:31.654022+0000","last_unstale":"2023-10-11T11:19:02.769176+0000","last_undegraded":"2023-10-11T11:15:31.646946+0000","last_fullsized":"2023-10-11T11:15:31.646751+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:36.739637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:36.739637+0000","last_clean_scrub_stamp":"2023-10-11T10:43:36.739637+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T11:33:33.696447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":520,"reported_epoch":500,"state":"active+clean+remapped","last_fresh":"2023-10-11T11:19:03.429429+0000","last_change":"2023-10-11T11:15:33.660200+0000","last_active":"2023-10-11T11:19:03.429429+0000","last_peered":"2023-10-11T11:19:03.429429+0000","last_clean":"2023-10-11T11:19:03.429429+0000","last_became_active":"2023-10-11T11:15:33.659619+0000","last_became_peered":"2023-10-11T11:15:33.659619+0000","last_unstale":"2023-10-11T11:19:03.429429+0000","last_undegraded":"2023-10-11T11:19:03.429429+0000","last_fullsized":"2023-10-11T11:19:03.429429+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T17:21:16.866392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":517,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-10-11T11:19:02.775128+0000","last_change":"2023-10-11T11:15:31.670813+0000","last_active":"2023-10-11T11:19:02.775128+0000","last_peered":"2023-10-11T11:19:02.775128+0000","last_clean":"2023-10-11T11:15:27.638219+0000","last_became_active":"2023-10-11T11:15:31.670813+0000","last_became_peered":"2023-10-11T11:15:31.670813+0000","last_unstale":"2023-10-11T11:19:02.775128+0000","last_undegraded":"2023-10-11T11:19:02.775128+0000","last_fullsized":"2023-10-11T11:15:31.658785+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:41:50.872346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":516,"reported_epoch":499,"state":"active+undersized","last_fresh":"2023-10-11T11:19:00.631668+0000","last_change":"2023-10-11T11:15:31.666378+0000","last_active":"2023-10-11T11:19:00.631668+0000","last_peered":"2023-10-11T11:19:00.631668+0000","last_clean":"2023-10-11T11:15:27.638303+0000","last_became_active":"2023-10-11T11:15:31.666378+0000","last_became_peered":"2023-10-11T11:15:31.666378+0000","last_unstale":"2023-10-11T11:19:00.631668+0000","last_undegraded":"2023-10-11T11:19:00.631668+0000","last_fullsized":"2023-10-11T11:15:31.649186+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T15:35:19.331209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":516,"reported_epoch":499,"state":"active+undersized","last_fresh":"2023-10-11T11:19:00.636960+0000","last_change":"2023-10-11T11:15:31.679465+0000","last_active":"2023-10-11T11:19:00.636960+0000","last_peered":"2023-10-11T11:19:00.636960+0000","last_clean":"2023-10-11T11:15:27.638749+0000","last_became_active":"2023-10-11T11:15:31.679465+0000","last_became_peered":"2023-10-11T11:15:31.679465+0000","last_unstale":"2023-10-11T11:19:00.636960+0000","last_undegraded":"2023-10-11T11:19:00.636960+0000","last_fullsized":"2023-10-11T11:15:31.659135+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T16:03:42.656692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":11,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738807,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":785348,"kb_used_data":5812,"kb_used_omap":96,"kb_used_meta":779423,"kb_avail":93586492,"statfs":{"total":96636764160,"available":95832567808,"internally_reserved":0,"allocated":5951488,"data_stored":3106918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":98609,"internal_metadata":798129871},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 11:14:37 2023","interfaces":[{"interface":"back","average":{"1min":0.314,"5min":0.316,"15min":0.443},"min":{"1min":0.24399999999999999,"5min":0.14799999999999999,"15min":0.14599999999999999},"max":{"1min":0.48499999999999999,"5min":1.712,"15min":9.6400000000000006},"last":0.31},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.35499999999999998,"15min":0.41299999999999998},"min":{"1min":0.20899999999999999,"5min":0.17199999999999999,"15min":0.16400000000000001},"max":{"1min":3.8180000000000001,"5min":3.8180000000000001,"15min":6.3200000000000003},"last":0.26500000000000001}]},{"osd":1,"last update":"Wed Oct 11 11:19:40 2023","interfaces":[{"interface":"back","average":{"1min":0.29599999999999999,"5min":0.28499999999999998,"15min":0.32300000000000001},"min":{"1min":0.17699999999999999,"5min":0.154,"15min":0.154},"max":{"1min":0.40600000000000003,"5min":0.41999999999999998,"15min":2.9540000000000002},"last":0.27000000000000002},{"interface":"front","average":{"1min":0.317,"5min":0.31,"15min":0.36199999999999999},"min":{"1min":0.20899999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.45000000000000001,"5min":0.49099999999999999,"15min":3.2719999999999998},"last":0.30599999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738756,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1484496,"kb_used_data":406792,"kb_used_omap":107,"kb_used_meta":1077588,"kb_avail":92887344,"statfs":{"total":96636764160,"available":95116640256,"internally_reserved":0,"allocated":416555008,"data_stored":413648921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110309,"internal_metadata":1103450395},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Oct 11 11:14:30 2023","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.35399999999999998,"15min":0.40600000000000003},"min":{"1min":0.18099999999999999,"5min":0.155,"15min":0.14899999999999999},"max":{"1min":1.4319999999999999,"5min":2.3460000000000001,"15min":6.593},"last":0.26200000000000001},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.39100000000000001,"15min":0.48199999999999998},"min":{"1min":0.189,"5min":0.189,"15min":0.183},"max":{"1min":2.2120000000000002,"5min":2.4089999999999998,"15min":6.5960000000000001},"last":0.316}]},{"osd":2,"last update":"Wed Oct 11 11:14:30 2023","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.35499999999999998,"15min":0.47099999999999997},"min":{"1min":0.20300000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":1.2490000000000001,"5min":2.665,"15min":8.4730000000000008},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.375,"15min":0.442},"min":{"1min":0.23200000000000001,"5min":0.20499999999999999,"15min":0.19600000000000001},"max":{"1min":1.28,"5min":2.6230000000000002,"15min":7.0129999999999999},"last":0.33600000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738808,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":826440,"kb_used_data":6392,"kb_used_omap":88,"kb_used_meta":819943,"kb_avail":93545400,"statfs":{"total":96636764160,"available":95790489600,"internally_reserved":0,"allocated":6545408,"data_stored":3697286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":90523,"internal_metadata":839622245},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 11:14:33 2023","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.42799999999999999,"15min":0.41099999999999998},"min":{"1min":0.151,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":1.2809999999999999,"5min":3.5739999999999998,"15min":8.2479999999999993},"last":0.223},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.48299999999999998,"15min":0.49199999999999999},"min":{"1min":0.20699999999999999,"5min":0.182,"15min":0.17999999999999999},"max":{"1min":1.323,"5min":3.6080000000000001,"15min":8.2870000000000008},"last":0.28399999999999997}]},{"osd":2,"last update":"Wed Oct 11 11:19:41 2023","interfaces":[{"interface":"back","average":{"1min":0.314,"5min":0.35399999999999998,"15min":0.35899999999999999},"min":{"1min":0.186,"5min":0.14699999999999999,"15min":0.14699999999999999},"max":{"1min":0.45500000000000002,"5min":7.4829999999999997,"15min":10.617000000000001},"last":0.312},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.39400000000000002,"15min":0.35099999999999998},"min":{"1min":0.21199999999999999,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.49099999999999999,"5min":7.5350000000000001,"15min":7.5350000000000001},"last":0.34000000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-11T11:19:57.966 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-10-11T11:19:57.967 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 517, 'reported_epoch': 500, 'state': 'active+undersized', 'last_fresh': '2023-10-11T11:19:02.772828+0000', 'last_change': '2023-10-11T11:15:31.661747+0000', 'last_active': '2023-10-11T11:19:02.772828+0000', 'last_peered': '2023-10-11T11:19:02.772828+0000', 'last_clean': '2023-10-11T11:15:27.638215+0000', 'last_became_active': '2023-10-11T11:15:31.661747+0000', 'last_became_peered': '2023-10-11T11:15:31.661747+0000', 'last_unstale': '2023-10-11T11:19:02.772828+0000', 'last_undegraded': '2023-10-11T11:19:02.772828+0000', 'last_fullsized': '2023-10-11T11:15:31.649336+0000', 'mapping_epoch': 480, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_clean_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-12T22:15:25.047039+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-10-11T11:19:57.967 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-10-11T11:19:57.967 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 29, 'reported_epoch': 500, 'state': 'active+undersized', 'last_fresh': '2023-10-11T11:19:02.774213+0000', 'last_change': '2023-10-11T11:15:31.671363+0000', 'last_active': '2023-10-11T11:19:02.774213+0000', 'last_peered': '2023-10-11T11:19:02.774213+0000', 'last_clean': '2023-10-11T10:43:40.751300+0000', 'last_became_active': '2023-10-11T11:15:31.671363+0000', 'last_became_peered': '2023-10-11T11:15:31.671363+0000', 'last_unstale': '2023-10-11T11:19:02.774213+0000', 'last_undegraded': '2023-10-11T11:19:02.774213+0000', 'last_fullsized': '2023-10-11T11:15:31.652001+0000', 'mapping_epoch': 480, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_clean_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-12T21:01:53.884964+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-10-11T11:19:57.968 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-10-11T11:19:57.968 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "11'97", 'reported_seq': 648, 'reported_epoch': 500, 'state': 'active+undersized+degraded', 'last_fresh': '2023-10-11T11:19:02.769176+0000', 'last_change': '2023-10-11T11:15:31.654022+0000', 'last_active': '2023-10-11T11:19:02.769176+0000', 'last_peered': '2023-10-11T11:19:02.769176+0000', 'last_clean': '2023-10-11T11:15:27.636884+0000', 'last_became_active': '2023-10-11T11:15:31.654022+0000', 'last_became_peered': '2023-10-11T11:15:31.654022+0000', 'last_unstale': '2023-10-11T11:19:02.769176+0000', 'last_undegraded': '2023-10-11T11:15:31.646946+0000', 'last_fullsized': '2023-10-11T11:15:31.646751+0000', 'mapping_epoch': 480, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-11T10:43:36.739637+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-11T10:43:36.739637+0000', 'last_clean_scrub_stamp': '2023-10-11T10:43:36.739637+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-12T11:33:33.696447+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-10-11T11:19:57.968 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.0 is not active+clean 2023-10-11T11:19:57.969 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 520, 'reported_epoch': 500, 'state': 'active+clean+remapped', 'last_fresh': '2023-10-11T11:19:03.429429+0000', 'last_change': '2023-10-11T11:15:33.660200+0000', 'last_active': '2023-10-11T11:19:03.429429+0000', 'last_peered': '2023-10-11T11:19:03.429429+0000', 'last_clean': '2023-10-11T11:19:03.429429+0000', 'last_became_active': '2023-10-11T11:15:33.659619+0000', 'last_became_peered': '2023-10-11T11:15:33.659619+0000', 'last_unstale': '2023-10-11T11:19:03.429429+0000', 'last_undegraded': '2023-10-11T11:19:03.429429+0000', 'last_fullsized': '2023-10-11T11:19:03.429429+0000', 'mapping_epoch': 482, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 483, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_clean_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-12T17:21:16.866392+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2, 1], 'avail_no_missing': ['2', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-10-11T11:19:57.969 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-10-11T11:19:57.969 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 517, 'reported_epoch': 500, 'state': 'active+undersized', 'last_fresh': '2023-10-11T11:19:02.775128+0000', 'last_change': '2023-10-11T11:15:31.670813+0000', 'last_active': '2023-10-11T11:19:02.775128+0000', 'last_peered': '2023-10-11T11:19:02.775128+0000', 'last_clean': '2023-10-11T11:15:27.638219+0000', 'last_became_active': '2023-10-11T11:15:31.670813+0000', 'last_became_peered': '2023-10-11T11:15:31.670813+0000', 'last_unstale': '2023-10-11T11:19:02.775128+0000', 'last_undegraded': '2023-10-11T11:19:02.775128+0000', 'last_fullsized': '2023-10-11T11:15:31.658785+0000', 'mapping_epoch': 480, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_clean_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-12T21:41:50.872346+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-10-11T11:19:57.970 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-10-11T11:19:57.970 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 516, 'reported_epoch': 499, 'state': 'active+undersized', 'last_fresh': '2023-10-11T11:19:00.631668+0000', 'last_change': '2023-10-11T11:15:31.666378+0000', 'last_active': '2023-10-11T11:19:00.631668+0000', 'last_peered': '2023-10-11T11:19:00.631668+0000', 'last_clean': '2023-10-11T11:15:27.638303+0000', 'last_became_active': '2023-10-11T11:15:31.666378+0000', 'last_became_peered': '2023-10-11T11:15:31.666378+0000', 'last_unstale': '2023-10-11T11:19:00.631668+0000', 'last_undegraded': '2023-10-11T11:19:00.631668+0000', 'last_fullsized': '2023-10-11T11:15:31.649186+0000', 'mapping_epoch': 480, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_clean_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-12T15:35:19.331209+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-10-11T11:19:57.970 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-10-11T11:19:57.970 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 516, 'reported_epoch': 499, 'state': 'active+undersized', 'last_fresh': '2023-10-11T11:19:00.636960+0000', 'last_change': '2023-10-11T11:15:31.679465+0000', 'last_active': '2023-10-11T11:19:00.636960+0000', 'last_peered': '2023-10-11T11:19:00.636960+0000', 'last_clean': '2023-10-11T11:15:27.638749+0000', 'last_became_active': '2023-10-11T11:15:31.679465+0000', 'last_became_peered': '2023-10-11T11:15:31.679465+0000', 'last_unstale': '2023-10-11T11:19:00.636960+0000', 'last_undegraded': '2023-10-11T11:19:00.636960+0000', 'last_fullsized': '2023-10-11T11:15:31.659135+0000', 'mapping_epoch': 480, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_clean_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-12T16:03:42.656692+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-10-11T11:19:57.971 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-10-11T11:19:58.368 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T11:19:58.369 INFO:teuthology.orchestra.run.smithi111.stdout:{"fsid":"022ce6f2-9a5b-45ad-8b77-a1332d92f228","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2193,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":500,"num_osds":3,"num_up_osds":2,"osd_up_since":1697022930,"num_in_osds":3,"osd_in_since":1697021005,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":2},{"state_name":"active+undersized+degraded","count":1},{"state_name":"active+clean+remapped","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3170594816,"bytes_avail":286739697664,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":36,"modified":"2023-10-11T11:19:04.644408+0000","services":{}},"progress_events":{}} 2023-10-11T11:19:58.383 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-10-11T11:19:58.383 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-10-11T11:19:58.700 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T11:19:58.702 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2023-10-11T11:19:58.713 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":1590,"stamp":"2023-10-11T11:19:58.657408+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":11,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":107,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3096284,"kb_used_data":418996,"kb_used_omap":292,"kb_used_meta":2676955,"kb_avail":280019236,"statfs":{"total":289910292480,"available":286739697664,"internally_reserved":0,"allocated":429051904,"data_stored":420453125,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299441,"internal_metadata":2741202511},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002924"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":517,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-10-11T11:19:02.772828+0000","last_change":"2023-10-11T11:15:31.661747+0000","last_active":"2023-10-11T11:19:02.772828+0000","last_peered":"2023-10-11T11:19:02.772828+0000","last_clean":"2023-10-11T11:15:27.638215+0000","last_became_active":"2023-10-11T11:15:31.661747+0000","last_became_peered":"2023-10-11T11:15:31.661747+0000","last_unstale":"2023-10-11T11:19:02.772828+0000","last_undegraded":"2023-10-11T11:19:02.772828+0000","last_fullsized":"2023-10-11T11:15:31.649336+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:15:25.047039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":29,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-10-11T11:19:02.774213+0000","last_change":"2023-10-11T11:15:31.671363+0000","last_active":"2023-10-11T11:19:02.774213+0000","last_peered":"2023-10-11T11:19:02.774213+0000","last_clean":"2023-10-11T10:43:40.751300+0000","last_became_active":"2023-10-11T11:15:31.671363+0000","last_became_peered":"2023-10-11T11:15:31.671363+0000","last_unstale":"2023-10-11T11:19:02.774213+0000","last_undegraded":"2023-10-11T11:19:02.774213+0000","last_fullsized":"2023-10-11T11:15:31.652001+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:01:53.884964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":512,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-10-11T11:19:03.429640+0000","last_change":"2023-10-11T10:43:41.772781+0000","last_active":"2023-10-11T11:19:03.429640+0000","last_peered":"2023-10-11T11:19:03.429640+0000","last_clean":"2023-10-11T11:19:03.429640+0000","last_became_active":"2023-10-11T10:43:41.772443+0000","last_became_peered":"2023-10-11T10:43:41.772443+0000","last_unstale":"2023-10-11T11:19:03.429640+0000","last_undegraded":"2023-10-11T11:19:03.429640+0000","last_fullsized":"2023-10-11T11:19:03.429640+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T14:15:30.547284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":513,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-10-11T11:19:02.772211+0000","last_change":"2023-10-11T10:43:41.772615+0000","last_active":"2023-10-11T11:19:02.772211+0000","last_peered":"2023-10-11T11:19:02.772211+0000","last_clean":"2023-10-11T11:19:02.772211+0000","last_became_active":"2023-10-11T10:43:41.772290+0000","last_became_peered":"2023-10-11T10:43:41.772290+0000","last_unstale":"2023-10-11T11:19:02.772211+0000","last_undegraded":"2023-10-11T11:19:02.772211+0000","last_fullsized":"2023-10-11T11:19:02.772211+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:26:06.909963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":648,"reported_epoch":500,"state":"active+undersized+degraded","last_fresh":"2023-10-11T11:19:02.769176+0000","last_change":"2023-10-11T11:15:31.654022+0000","last_active":"2023-10-11T11:19:02.769176+0000","last_peered":"2023-10-11T11:19:02.769176+0000","last_clean":"2023-10-11T11:15:27.636884+0000","last_became_active":"2023-10-11T11:15:31.654022+0000","last_became_peered":"2023-10-11T11:15:31.654022+0000","last_unstale":"2023-10-11T11:19:02.769176+0000","last_undegraded":"2023-10-11T11:15:31.646946+0000","last_fullsized":"2023-10-11T11:15:31.646751+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:36.739637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:36.739637+0000","last_clean_scrub_stamp":"2023-10-11T10:43:36.739637+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T11:33:33.696447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":520,"reported_epoch":500,"state":"active+clean+remapped","last_fresh":"2023-10-11T11:19:03.429429+0000","last_change":"2023-10-11T11:15:33.660200+0000","last_active":"2023-10-11T11:19:03.429429+0000","last_peered":"2023-10-11T11:19:03.429429+0000","last_clean":"2023-10-11T11:19:03.429429+0000","last_became_active":"2023-10-11T11:15:33.659619+0000","last_became_peered":"2023-10-11T11:15:33.659619+0000","last_unstale":"2023-10-11T11:19:03.429429+0000","last_undegraded":"2023-10-11T11:19:03.429429+0000","last_fullsized":"2023-10-11T11:19:03.429429+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T17:21:16.866392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":517,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-10-11T11:19:02.775128+0000","last_change":"2023-10-11T11:15:31.670813+0000","last_active":"2023-10-11T11:19:02.775128+0000","last_peered":"2023-10-11T11:19:02.775128+0000","last_clean":"2023-10-11T11:15:27.638219+0000","last_became_active":"2023-10-11T11:15:31.670813+0000","last_became_peered":"2023-10-11T11:15:31.670813+0000","last_unstale":"2023-10-11T11:19:02.775128+0000","last_undegraded":"2023-10-11T11:19:02.775128+0000","last_fullsized":"2023-10-11T11:15:31.658785+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:41:50.872346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":516,"reported_epoch":499,"state":"active+undersized","last_fresh":"2023-10-11T11:19:00.631668+0000","last_change":"2023-10-11T11:15:31.666378+0000","last_active":"2023-10-11T11:19:00.631668+0000","last_peered":"2023-10-11T11:19:00.631668+0000","last_clean":"2023-10-11T11:15:27.638303+0000","last_became_active":"2023-10-11T11:15:31.666378+0000","last_became_peered":"2023-10-11T11:15:31.666378+0000","last_unstale":"2023-10-11T11:19:00.631668+0000","last_undegraded":"2023-10-11T11:19:00.631668+0000","last_fullsized":"2023-10-11T11:15:31.649186+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T15:35:19.331209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":516,"reported_epoch":499,"state":"active+undersized","last_fresh":"2023-10-11T11:19:00.636960+0000","last_change":"2023-10-11T11:15:31.679465+0000","last_active":"2023-10-11T11:19:00.636960+0000","last_peered":"2023-10-11T11:19:00.636960+0000","last_clean":"2023-10-11T11:15:27.638749+0000","last_became_active":"2023-10-11T11:15:31.679465+0000","last_became_peered":"2023-10-11T11:15:31.679465+0000","last_unstale":"2023-10-11T11:19:00.636960+0000","last_undegraded":"2023-10-11T11:19:00.636960+0000","last_fullsized":"2023-10-11T11:15:31.659135+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T16:03:42.656692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":11,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738807,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":785348,"kb_used_data":5812,"kb_used_omap":96,"kb_used_meta":779423,"kb_avail":93586492,"statfs":{"total":96636764160,"available":95832567808,"internally_reserved":0,"allocated":5951488,"data_stored":3106918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":98609,"internal_metadata":798129871},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 11:14:37 2023","interfaces":[{"interface":"back","average":{"1min":0.314,"5min":0.316,"15min":0.443},"min":{"1min":0.24399999999999999,"5min":0.14799999999999999,"15min":0.14599999999999999},"max":{"1min":0.48499999999999999,"5min":1.712,"15min":9.6400000000000006},"last":0.31},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.35499999999999998,"15min":0.41299999999999998},"min":{"1min":0.20899999999999999,"5min":0.17199999999999999,"15min":0.16400000000000001},"max":{"1min":3.8180000000000001,"5min":3.8180000000000001,"15min":6.3200000000000003},"last":0.26500000000000001}]},{"osd":1,"last update":"Wed Oct 11 11:19:40 2023","interfaces":[{"interface":"back","average":{"1min":0.29599999999999999,"5min":0.28499999999999998,"15min":0.32300000000000001},"min":{"1min":0.17699999999999999,"5min":0.154,"15min":0.154},"max":{"1min":0.40600000000000003,"5min":0.41999999999999998,"15min":2.9540000000000002},"last":0.27000000000000002},{"interface":"front","average":{"1min":0.317,"5min":0.31,"15min":0.36199999999999999},"min":{"1min":0.20899999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.45000000000000001,"5min":0.49099999999999999,"15min":3.2719999999999998},"last":0.30599999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738756,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1484496,"kb_used_data":406792,"kb_used_omap":107,"kb_used_meta":1077588,"kb_avail":92887344,"statfs":{"total":96636764160,"available":95116640256,"internally_reserved":0,"allocated":416555008,"data_stored":413648921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110309,"internal_metadata":1103450395},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Oct 11 11:14:30 2023","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.35399999999999998,"15min":0.40600000000000003},"min":{"1min":0.18099999999999999,"5min":0.155,"15min":0.14899999999999999},"max":{"1min":1.4319999999999999,"5min":2.3460000000000001,"15min":6.593},"last":0.26200000000000001},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.39100000000000001,"15min":0.48199999999999998},"min":{"1min":0.189,"5min":0.189,"15min":0.183},"max":{"1min":2.2120000000000002,"5min":2.4089999999999998,"15min":6.5960000000000001},"last":0.316}]},{"osd":2,"last update":"Wed Oct 11 11:14:30 2023","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.35499999999999998,"15min":0.47099999999999997},"min":{"1min":0.20300000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":1.2490000000000001,"5min":2.665,"15min":8.4730000000000008},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.375,"15min":0.442},"min":{"1min":0.23200000000000001,"5min":0.20499999999999999,"15min":0.19600000000000001},"max":{"1min":1.28,"5min":2.6230000000000002,"15min":7.0129999999999999},"last":0.33600000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738808,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":826440,"kb_used_data":6392,"kb_used_omap":88,"kb_used_meta":819943,"kb_avail":93545400,"statfs":{"total":96636764160,"available":95790489600,"internally_reserved":0,"allocated":6545408,"data_stored":3697286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":90523,"internal_metadata":839622245},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 11:14:33 2023","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.42799999999999999,"15min":0.41099999999999998},"min":{"1min":0.151,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":1.2809999999999999,"5min":3.5739999999999998,"15min":8.2479999999999993},"last":0.223},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.48299999999999998,"15min":0.49199999999999999},"min":{"1min":0.20699999999999999,"5min":0.182,"15min":0.17999999999999999},"max":{"1min":1.323,"5min":3.6080000000000001,"15min":8.2870000000000008},"last":0.28399999999999997}]},{"osd":2,"last update":"Wed Oct 11 11:19:41 2023","interfaces":[{"interface":"back","average":{"1min":0.314,"5min":0.35399999999999998,"15min":0.35899999999999999},"min":{"1min":0.186,"5min":0.14699999999999999,"15min":0.14699999999999999},"max":{"1min":0.45500000000000002,"5min":7.4829999999999997,"15min":10.617000000000001},"last":0.312},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.39400000000000002,"15min":0.35099999999999998},"min":{"1min":0.21199999999999999,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.49099999999999999,"5min":7.5350000000000001,"15min":7.5350000000000001},"last":0.34000000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-11T11:20:01.716 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-10-11T11:20:02.049 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T11:20:02.051 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2023-10-11T11:20:02.062 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":1591,"stamp":"2023-10-11T11:20:00.657738+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":11,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":107,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3096284,"kb_used_data":418996,"kb_used_omap":292,"kb_used_meta":2676955,"kb_avail":280019236,"statfs":{"total":289910292480,"available":286739697664,"internally_reserved":0,"allocated":429051904,"data_stored":420453125,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299441,"internal_metadata":2741202511},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002853"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":517,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-10-11T11:19:02.772828+0000","last_change":"2023-10-11T11:15:31.661747+0000","last_active":"2023-10-11T11:19:02.772828+0000","last_peered":"2023-10-11T11:19:02.772828+0000","last_clean":"2023-10-11T11:15:27.638215+0000","last_became_active":"2023-10-11T11:15:31.661747+0000","last_became_peered":"2023-10-11T11:15:31.661747+0000","last_unstale":"2023-10-11T11:19:02.772828+0000","last_undegraded":"2023-10-11T11:19:02.772828+0000","last_fullsized":"2023-10-11T11:15:31.649336+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:15:25.047039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":29,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-10-11T11:19:02.774213+0000","last_change":"2023-10-11T11:15:31.671363+0000","last_active":"2023-10-11T11:19:02.774213+0000","last_peered":"2023-10-11T11:19:02.774213+0000","last_clean":"2023-10-11T10:43:40.751300+0000","last_became_active":"2023-10-11T11:15:31.671363+0000","last_became_peered":"2023-10-11T11:15:31.671363+0000","last_unstale":"2023-10-11T11:19:02.774213+0000","last_undegraded":"2023-10-11T11:19:02.774213+0000","last_fullsized":"2023-10-11T11:15:31.652001+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:01:53.884964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":512,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-10-11T11:19:03.429640+0000","last_change":"2023-10-11T10:43:41.772781+0000","last_active":"2023-10-11T11:19:03.429640+0000","last_peered":"2023-10-11T11:19:03.429640+0000","last_clean":"2023-10-11T11:19:03.429640+0000","last_became_active":"2023-10-11T10:43:41.772443+0000","last_became_peered":"2023-10-11T10:43:41.772443+0000","last_unstale":"2023-10-11T11:19:03.429640+0000","last_undegraded":"2023-10-11T11:19:03.429640+0000","last_fullsized":"2023-10-11T11:19:03.429640+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T14:15:30.547284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":513,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-10-11T11:19:02.772211+0000","last_change":"2023-10-11T10:43:41.772615+0000","last_active":"2023-10-11T11:19:02.772211+0000","last_peered":"2023-10-11T11:19:02.772211+0000","last_clean":"2023-10-11T11:19:02.772211+0000","last_became_active":"2023-10-11T10:43:41.772290+0000","last_became_peered":"2023-10-11T10:43:41.772290+0000","last_unstale":"2023-10-11T11:19:02.772211+0000","last_undegraded":"2023-10-11T11:19:02.772211+0000","last_fullsized":"2023-10-11T11:19:02.772211+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:26:06.909963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":648,"reported_epoch":500,"state":"active+undersized+degraded","last_fresh":"2023-10-11T11:19:02.769176+0000","last_change":"2023-10-11T11:15:31.654022+0000","last_active":"2023-10-11T11:19:02.769176+0000","last_peered":"2023-10-11T11:19:02.769176+0000","last_clean":"2023-10-11T11:15:27.636884+0000","last_became_active":"2023-10-11T11:15:31.654022+0000","last_became_peered":"2023-10-11T11:15:31.654022+0000","last_unstale":"2023-10-11T11:19:02.769176+0000","last_undegraded":"2023-10-11T11:15:31.646946+0000","last_fullsized":"2023-10-11T11:15:31.646751+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:36.739637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:36.739637+0000","last_clean_scrub_stamp":"2023-10-11T10:43:36.739637+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T11:33:33.696447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":520,"reported_epoch":500,"state":"active+clean+remapped","last_fresh":"2023-10-11T11:19:03.429429+0000","last_change":"2023-10-11T11:15:33.660200+0000","last_active":"2023-10-11T11:19:03.429429+0000","last_peered":"2023-10-11T11:19:03.429429+0000","last_clean":"2023-10-11T11:19:03.429429+0000","last_became_active":"2023-10-11T11:15:33.659619+0000","last_became_peered":"2023-10-11T11:15:33.659619+0000","last_unstale":"2023-10-11T11:19:03.429429+0000","last_undegraded":"2023-10-11T11:19:03.429429+0000","last_fullsized":"2023-10-11T11:19:03.429429+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T17:21:16.866392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":517,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-10-11T11:19:02.775128+0000","last_change":"2023-10-11T11:15:31.670813+0000","last_active":"2023-10-11T11:19:02.775128+0000","last_peered":"2023-10-11T11:19:02.775128+0000","last_clean":"2023-10-11T11:15:27.638219+0000","last_became_active":"2023-10-11T11:15:31.670813+0000","last_became_peered":"2023-10-11T11:15:31.670813+0000","last_unstale":"2023-10-11T11:19:02.775128+0000","last_undegraded":"2023-10-11T11:19:02.775128+0000","last_fullsized":"2023-10-11T11:15:31.658785+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:41:50.872346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":516,"reported_epoch":499,"state":"active+undersized","last_fresh":"2023-10-11T11:19:00.631668+0000","last_change":"2023-10-11T11:15:31.666378+0000","last_active":"2023-10-11T11:19:00.631668+0000","last_peered":"2023-10-11T11:19:00.631668+0000","last_clean":"2023-10-11T11:15:27.638303+0000","last_became_active":"2023-10-11T11:15:31.666378+0000","last_became_peered":"2023-10-11T11:15:31.666378+0000","last_unstale":"2023-10-11T11:19:00.631668+0000","last_undegraded":"2023-10-11T11:19:00.631668+0000","last_fullsized":"2023-10-11T11:15:31.649186+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T15:35:19.331209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":516,"reported_epoch":499,"state":"active+undersized","last_fresh":"2023-10-11T11:19:00.636960+0000","last_change":"2023-10-11T11:15:31.679465+0000","last_active":"2023-10-11T11:19:00.636960+0000","last_peered":"2023-10-11T11:19:00.636960+0000","last_clean":"2023-10-11T11:15:27.638749+0000","last_became_active":"2023-10-11T11:15:31.679465+0000","last_became_peered":"2023-10-11T11:15:31.679465+0000","last_unstale":"2023-10-11T11:19:00.636960+0000","last_undegraded":"2023-10-11T11:19:00.636960+0000","last_fullsized":"2023-10-11T11:15:31.659135+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T16:03:42.656692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":11,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738808,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":785348,"kb_used_data":5812,"kb_used_omap":96,"kb_used_meta":779423,"kb_avail":93586492,"statfs":{"total":96636764160,"available":95832567808,"internally_reserved":0,"allocated":5951488,"data_stored":3106918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":98609,"internal_metadata":798129871},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 11:14:37 2023","interfaces":[{"interface":"back","average":{"1min":0.314,"5min":0.316,"15min":0.443},"min":{"1min":0.24399999999999999,"5min":0.14799999999999999,"15min":0.14599999999999999},"max":{"1min":0.48499999999999999,"5min":1.712,"15min":9.6400000000000006},"last":0.31},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.35499999999999998,"15min":0.41299999999999998},"min":{"1min":0.20899999999999999,"5min":0.17199999999999999,"15min":0.16400000000000001},"max":{"1min":3.8180000000000001,"5min":3.8180000000000001,"15min":6.3200000000000003},"last":0.26500000000000001}]},{"osd":1,"last update":"Wed Oct 11 11:19:40 2023","interfaces":[{"interface":"back","average":{"1min":0.29599999999999999,"5min":0.28499999999999998,"15min":0.32300000000000001},"min":{"1min":0.17699999999999999,"5min":0.154,"15min":0.154},"max":{"1min":0.40600000000000003,"5min":0.41999999999999998,"15min":2.9540000000000002},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.317,"5min":0.31,"15min":0.36199999999999999},"min":{"1min":0.20899999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.45000000000000001,"5min":0.49099999999999999,"15min":3.2719999999999998},"last":0.42399999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738756,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1484496,"kb_used_data":406792,"kb_used_omap":107,"kb_used_meta":1077588,"kb_avail":92887344,"statfs":{"total":96636764160,"available":95116640256,"internally_reserved":0,"allocated":416555008,"data_stored":413648921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110309,"internal_metadata":1103450395},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Oct 11 11:14:30 2023","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.35399999999999998,"15min":0.40600000000000003},"min":{"1min":0.18099999999999999,"5min":0.155,"15min":0.14899999999999999},"max":{"1min":1.4319999999999999,"5min":2.3460000000000001,"15min":6.593},"last":0.26200000000000001},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.39100000000000001,"15min":0.48199999999999998},"min":{"1min":0.189,"5min":0.189,"15min":0.183},"max":{"1min":2.2120000000000002,"5min":2.4089999999999998,"15min":6.5960000000000001},"last":0.316}]},{"osd":2,"last update":"Wed Oct 11 11:14:30 2023","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.35499999999999998,"15min":0.47099999999999997},"min":{"1min":0.20300000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":1.2490000000000001,"5min":2.665,"15min":8.4730000000000008},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.375,"15min":0.442},"min":{"1min":0.23200000000000001,"5min":0.20499999999999999,"15min":0.19600000000000001},"max":{"1min":1.28,"5min":2.6230000000000002,"15min":7.0129999999999999},"last":0.33600000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738809,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":826440,"kb_used_data":6392,"kb_used_omap":88,"kb_used_meta":819943,"kb_avail":93545400,"statfs":{"total":96636764160,"available":95790489600,"internally_reserved":0,"allocated":6545408,"data_stored":3697286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":90523,"internal_metadata":839622245},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 11:14:33 2023","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.42799999999999999,"15min":0.41099999999999998},"min":{"1min":0.151,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":1.2809999999999999,"5min":3.5739999999999998,"15min":8.2479999999999993},"last":0.223},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.48299999999999998,"15min":0.49199999999999999},"min":{"1min":0.20699999999999999,"5min":0.182,"15min":0.17999999999999999},"max":{"1min":1.323,"5min":3.6080000000000001,"15min":8.2870000000000008},"last":0.28399999999999997}]},{"osd":2,"last update":"Wed Oct 11 11:19:41 2023","interfaces":[{"interface":"back","average":{"1min":0.314,"5min":0.35399999999999998,"15min":0.35899999999999999},"min":{"1min":0.186,"5min":0.14699999999999999,"15min":0.14699999999999999},"max":{"1min":0.45500000000000002,"5min":7.4829999999999997,"15min":10.617000000000001},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.39400000000000002,"15min":0.35099999999999998},"min":{"1min":0.21199999999999999,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.49099999999999999,"5min":7.5350000000000001,"15min":7.5350000000000001},"last":0.307}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-11T11:20:02.064 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-10-11T11:20:02.350 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~270s 2023-10-11T11:20:02.379 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T11:20:02.380 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2023-10-11T11:20:02.392 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":1591,"stamp":"2023-10-11T11:20:00.657738+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":11,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":107,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3096284,"kb_used_data":418996,"kb_used_omap":292,"kb_used_meta":2676955,"kb_avail":280019236,"statfs":{"total":289910292480,"available":286739697664,"internally_reserved":0,"allocated":429051904,"data_stored":420453125,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299441,"internal_metadata":2741202511},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002853"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":517,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-10-11T11:19:02.772828+0000","last_change":"2023-10-11T11:15:31.661747+0000","last_active":"2023-10-11T11:19:02.772828+0000","last_peered":"2023-10-11T11:19:02.772828+0000","last_clean":"2023-10-11T11:15:27.638215+0000","last_became_active":"2023-10-11T11:15:31.661747+0000","last_became_peered":"2023-10-11T11:15:31.661747+0000","last_unstale":"2023-10-11T11:19:02.772828+0000","last_undegraded":"2023-10-11T11:19:02.772828+0000","last_fullsized":"2023-10-11T11:15:31.649336+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:15:25.047039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":29,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-10-11T11:19:02.774213+0000","last_change":"2023-10-11T11:15:31.671363+0000","last_active":"2023-10-11T11:19:02.774213+0000","last_peered":"2023-10-11T11:19:02.774213+0000","last_clean":"2023-10-11T10:43:40.751300+0000","last_became_active":"2023-10-11T11:15:31.671363+0000","last_became_peered":"2023-10-11T11:15:31.671363+0000","last_unstale":"2023-10-11T11:19:02.774213+0000","last_undegraded":"2023-10-11T11:19:02.774213+0000","last_fullsized":"2023-10-11T11:15:31.652001+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:01:53.884964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":512,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-10-11T11:19:03.429640+0000","last_change":"2023-10-11T10:43:41.772781+0000","last_active":"2023-10-11T11:19:03.429640+0000","last_peered":"2023-10-11T11:19:03.429640+0000","last_clean":"2023-10-11T11:19:03.429640+0000","last_became_active":"2023-10-11T10:43:41.772443+0000","last_became_peered":"2023-10-11T10:43:41.772443+0000","last_unstale":"2023-10-11T11:19:03.429640+0000","last_undegraded":"2023-10-11T11:19:03.429640+0000","last_fullsized":"2023-10-11T11:19:03.429640+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T14:15:30.547284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":513,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-10-11T11:19:02.772211+0000","last_change":"2023-10-11T10:43:41.772615+0000","last_active":"2023-10-11T11:19:02.772211+0000","last_peered":"2023-10-11T11:19:02.772211+0000","last_clean":"2023-10-11T11:19:02.772211+0000","last_became_active":"2023-10-11T10:43:41.772290+0000","last_became_peered":"2023-10-11T10:43:41.772290+0000","last_unstale":"2023-10-11T11:19:02.772211+0000","last_undegraded":"2023-10-11T11:19:02.772211+0000","last_fullsized":"2023-10-11T11:19:02.772211+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:26:06.909963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":648,"reported_epoch":500,"state":"active+undersized+degraded","last_fresh":"2023-10-11T11:19:02.769176+0000","last_change":"2023-10-11T11:15:31.654022+0000","last_active":"2023-10-11T11:19:02.769176+0000","last_peered":"2023-10-11T11:19:02.769176+0000","last_clean":"2023-10-11T11:15:27.636884+0000","last_became_active":"2023-10-11T11:15:31.654022+0000","last_became_peered":"2023-10-11T11:15:31.654022+0000","last_unstale":"2023-10-11T11:19:02.769176+0000","last_undegraded":"2023-10-11T11:15:31.646946+0000","last_fullsized":"2023-10-11T11:15:31.646751+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:36.739637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:36.739637+0000","last_clean_scrub_stamp":"2023-10-11T10:43:36.739637+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T11:33:33.696447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":520,"reported_epoch":500,"state":"active+clean+remapped","last_fresh":"2023-10-11T11:19:03.429429+0000","last_change":"2023-10-11T11:15:33.660200+0000","last_active":"2023-10-11T11:19:03.429429+0000","last_peered":"2023-10-11T11:19:03.429429+0000","last_clean":"2023-10-11T11:19:03.429429+0000","last_became_active":"2023-10-11T11:15:33.659619+0000","last_became_peered":"2023-10-11T11:15:33.659619+0000","last_unstale":"2023-10-11T11:19:03.429429+0000","last_undegraded":"2023-10-11T11:19:03.429429+0000","last_fullsized":"2023-10-11T11:19:03.429429+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T17:21:16.866392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":517,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-10-11T11:19:02.775128+0000","last_change":"2023-10-11T11:15:31.670813+0000","last_active":"2023-10-11T11:19:02.775128+0000","last_peered":"2023-10-11T11:19:02.775128+0000","last_clean":"2023-10-11T11:15:27.638219+0000","last_became_active":"2023-10-11T11:15:31.670813+0000","last_became_peered":"2023-10-11T11:15:31.670813+0000","last_unstale":"2023-10-11T11:19:02.775128+0000","last_undegraded":"2023-10-11T11:19:02.775128+0000","last_fullsized":"2023-10-11T11:15:31.658785+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:41:50.872346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":516,"reported_epoch":499,"state":"active+undersized","last_fresh":"2023-10-11T11:19:00.631668+0000","last_change":"2023-10-11T11:15:31.666378+0000","last_active":"2023-10-11T11:19:00.631668+0000","last_peered":"2023-10-11T11:19:00.631668+0000","last_clean":"2023-10-11T11:15:27.638303+0000","last_became_active":"2023-10-11T11:15:31.666378+0000","last_became_peered":"2023-10-11T11:15:31.666378+0000","last_unstale":"2023-10-11T11:19:00.631668+0000","last_undegraded":"2023-10-11T11:19:00.631668+0000","last_fullsized":"2023-10-11T11:15:31.649186+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T15:35:19.331209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":516,"reported_epoch":499,"state":"active+undersized","last_fresh":"2023-10-11T11:19:00.636960+0000","last_change":"2023-10-11T11:15:31.679465+0000","last_active":"2023-10-11T11:19:00.636960+0000","last_peered":"2023-10-11T11:19:00.636960+0000","last_clean":"2023-10-11T11:15:27.638749+0000","last_became_active":"2023-10-11T11:15:31.679465+0000","last_became_peered":"2023-10-11T11:15:31.679465+0000","last_unstale":"2023-10-11T11:19:00.636960+0000","last_undegraded":"2023-10-11T11:19:00.636960+0000","last_fullsized":"2023-10-11T11:15:31.659135+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T16:03:42.656692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":11,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738808,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":785348,"kb_used_data":5812,"kb_used_omap":96,"kb_used_meta":779423,"kb_avail":93586492,"statfs":{"total":96636764160,"available":95832567808,"internally_reserved":0,"allocated":5951488,"data_stored":3106918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":98609,"internal_metadata":798129871},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 11:14:37 2023","interfaces":[{"interface":"back","average":{"1min":0.314,"5min":0.316,"15min":0.443},"min":{"1min":0.24399999999999999,"5min":0.14799999999999999,"15min":0.14599999999999999},"max":{"1min":0.48499999999999999,"5min":1.712,"15min":9.6400000000000006},"last":0.31},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.35499999999999998,"15min":0.41299999999999998},"min":{"1min":0.20899999999999999,"5min":0.17199999999999999,"15min":0.16400000000000001},"max":{"1min":3.8180000000000001,"5min":3.8180000000000001,"15min":6.3200000000000003},"last":0.26500000000000001}]},{"osd":1,"last update":"Wed Oct 11 11:19:40 2023","interfaces":[{"interface":"back","average":{"1min":0.29599999999999999,"5min":0.28499999999999998,"15min":0.32300000000000001},"min":{"1min":0.17699999999999999,"5min":0.154,"15min":0.154},"max":{"1min":0.40600000000000003,"5min":0.41999999999999998,"15min":2.9540000000000002},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.317,"5min":0.31,"15min":0.36199999999999999},"min":{"1min":0.20899999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.45000000000000001,"5min":0.49099999999999999,"15min":3.2719999999999998},"last":0.42399999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738756,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1484496,"kb_used_data":406792,"kb_used_omap":107,"kb_used_meta":1077588,"kb_avail":92887344,"statfs":{"total":96636764160,"available":95116640256,"internally_reserved":0,"allocated":416555008,"data_stored":413648921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110309,"internal_metadata":1103450395},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Oct 11 11:14:30 2023","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.35399999999999998,"15min":0.40600000000000003},"min":{"1min":0.18099999999999999,"5min":0.155,"15min":0.14899999999999999},"max":{"1min":1.4319999999999999,"5min":2.3460000000000001,"15min":6.593},"last":0.26200000000000001},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.39100000000000001,"15min":0.48199999999999998},"min":{"1min":0.189,"5min":0.189,"15min":0.183},"max":{"1min":2.2120000000000002,"5min":2.4089999999999998,"15min":6.5960000000000001},"last":0.316}]},{"osd":2,"last update":"Wed Oct 11 11:14:30 2023","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.35499999999999998,"15min":0.47099999999999997},"min":{"1min":0.20300000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":1.2490000000000001,"5min":2.665,"15min":8.4730000000000008},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.375,"15min":0.442},"min":{"1min":0.23200000000000001,"5min":0.20499999999999999,"15min":0.19600000000000001},"max":{"1min":1.28,"5min":2.6230000000000002,"15min":7.0129999999999999},"last":0.33600000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738809,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":826440,"kb_used_data":6392,"kb_used_omap":88,"kb_used_meta":819943,"kb_avail":93545400,"statfs":{"total":96636764160,"available":95790489600,"internally_reserved":0,"allocated":6545408,"data_stored":3697286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":90523,"internal_metadata":839622245},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 11:14:33 2023","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.42799999999999999,"15min":0.41099999999999998},"min":{"1min":0.151,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":1.2809999999999999,"5min":3.5739999999999998,"15min":8.2479999999999993},"last":0.223},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.48299999999999998,"15min":0.49199999999999999},"min":{"1min":0.20699999999999999,"5min":0.182,"15min":0.17999999999999999},"max":{"1min":1.323,"5min":3.6080000000000001,"15min":8.2870000000000008},"last":0.28399999999999997}]},{"osd":2,"last update":"Wed Oct 11 11:19:41 2023","interfaces":[{"interface":"back","average":{"1min":0.314,"5min":0.35399999999999998,"15min":0.35899999999999999},"min":{"1min":0.186,"5min":0.14699999999999999,"15min":0.14699999999999999},"max":{"1min":0.45500000000000002,"5min":7.4829999999999997,"15min":10.617000000000001},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.39400000000000002,"15min":0.35099999999999998},"min":{"1min":0.21199999999999999,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.49099999999999999,"5min":7.5350000000000001,"15min":7.5350000000000001},"last":0.307}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-11T11:20:02.393 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-10-11T11:20:02.393 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 517, 'reported_epoch': 500, 'state': 'active+undersized', 'last_fresh': '2023-10-11T11:19:02.772828+0000', 'last_change': '2023-10-11T11:15:31.661747+0000', 'last_active': '2023-10-11T11:19:02.772828+0000', 'last_peered': '2023-10-11T11:19:02.772828+0000', 'last_clean': '2023-10-11T11:15:27.638215+0000', 'last_became_active': '2023-10-11T11:15:31.661747+0000', 'last_became_peered': '2023-10-11T11:15:31.661747+0000', 'last_unstale': '2023-10-11T11:19:02.772828+0000', 'last_undegraded': '2023-10-11T11:19:02.772828+0000', 'last_fullsized': '2023-10-11T11:15:31.649336+0000', 'mapping_epoch': 480, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_clean_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-12T22:15:25.047039+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-10-11T11:20:02.394 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-10-11T11:20:02.394 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 29, 'reported_epoch': 500, 'state': 'active+undersized', 'last_fresh': '2023-10-11T11:19:02.774213+0000', 'last_change': '2023-10-11T11:15:31.671363+0000', 'last_active': '2023-10-11T11:19:02.774213+0000', 'last_peered': '2023-10-11T11:19:02.774213+0000', 'last_clean': '2023-10-11T10:43:40.751300+0000', 'last_became_active': '2023-10-11T11:15:31.671363+0000', 'last_became_peered': '2023-10-11T11:15:31.671363+0000', 'last_unstale': '2023-10-11T11:19:02.774213+0000', 'last_undegraded': '2023-10-11T11:19:02.774213+0000', 'last_fullsized': '2023-10-11T11:15:31.652001+0000', 'mapping_epoch': 480, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_clean_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-12T21:01:53.884964+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-10-11T11:20:02.394 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-10-11T11:20:02.395 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "11'97", 'reported_seq': 648, 'reported_epoch': 500, 'state': 'active+undersized+degraded', 'last_fresh': '2023-10-11T11:19:02.769176+0000', 'last_change': '2023-10-11T11:15:31.654022+0000', 'last_active': '2023-10-11T11:19:02.769176+0000', 'last_peered': '2023-10-11T11:19:02.769176+0000', 'last_clean': '2023-10-11T11:15:27.636884+0000', 'last_became_active': '2023-10-11T11:15:31.654022+0000', 'last_became_peered': '2023-10-11T11:15:31.654022+0000', 'last_unstale': '2023-10-11T11:19:02.769176+0000', 'last_undegraded': '2023-10-11T11:15:31.646946+0000', 'last_fullsized': '2023-10-11T11:15:31.646751+0000', 'mapping_epoch': 480, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-11T10:43:36.739637+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-11T10:43:36.739637+0000', 'last_clean_scrub_stamp': '2023-10-11T10:43:36.739637+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-12T11:33:33.696447+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-10-11T11:20:02.395 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.0 is not active+clean 2023-10-11T11:20:02.395 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 520, 'reported_epoch': 500, 'state': 'active+clean+remapped', 'last_fresh': '2023-10-11T11:19:03.429429+0000', 'last_change': '2023-10-11T11:15:33.660200+0000', 'last_active': '2023-10-11T11:19:03.429429+0000', 'last_peered': '2023-10-11T11:19:03.429429+0000', 'last_clean': '2023-10-11T11:19:03.429429+0000', 'last_became_active': '2023-10-11T11:15:33.659619+0000', 'last_became_peered': '2023-10-11T11:15:33.659619+0000', 'last_unstale': '2023-10-11T11:19:03.429429+0000', 'last_undegraded': '2023-10-11T11:19:03.429429+0000', 'last_fullsized': '2023-10-11T11:19:03.429429+0000', 'mapping_epoch': 482, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 483, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_clean_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-12T17:21:16.866392+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2, 1], 'avail_no_missing': ['2', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-10-11T11:20:02.396 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-10-11T11:20:02.396 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 517, 'reported_epoch': 500, 'state': 'active+undersized', 'last_fresh': '2023-10-11T11:19:02.775128+0000', 'last_change': '2023-10-11T11:15:31.670813+0000', 'last_active': '2023-10-11T11:19:02.775128+0000', 'last_peered': '2023-10-11T11:19:02.775128+0000', 'last_clean': '2023-10-11T11:15:27.638219+0000', 'last_became_active': '2023-10-11T11:15:31.670813+0000', 'last_became_peered': '2023-10-11T11:15:31.670813+0000', 'last_unstale': '2023-10-11T11:19:02.775128+0000', 'last_undegraded': '2023-10-11T11:19:02.775128+0000', 'last_fullsized': '2023-10-11T11:15:31.658785+0000', 'mapping_epoch': 480, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_clean_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-12T21:41:50.872346+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-10-11T11:20:02.397 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-10-11T11:20:02.397 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 516, 'reported_epoch': 499, 'state': 'active+undersized', 'last_fresh': '2023-10-11T11:19:00.631668+0000', 'last_change': '2023-10-11T11:15:31.666378+0000', 'last_active': '2023-10-11T11:19:00.631668+0000', 'last_peered': '2023-10-11T11:19:00.631668+0000', 'last_clean': '2023-10-11T11:15:27.638303+0000', 'last_became_active': '2023-10-11T11:15:31.666378+0000', 'last_became_peered': '2023-10-11T11:15:31.666378+0000', 'last_unstale': '2023-10-11T11:19:00.631668+0000', 'last_undegraded': '2023-10-11T11:19:00.631668+0000', 'last_fullsized': '2023-10-11T11:15:31.649186+0000', 'mapping_epoch': 480, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_clean_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-12T15:35:19.331209+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-10-11T11:20:02.397 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-10-11T11:20:02.398 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 516, 'reported_epoch': 499, 'state': 'active+undersized', 'last_fresh': '2023-10-11T11:19:00.636960+0000', 'last_change': '2023-10-11T11:15:31.679465+0000', 'last_active': '2023-10-11T11:19:00.636960+0000', 'last_peered': '2023-10-11T11:19:00.636960+0000', 'last_clean': '2023-10-11T11:15:27.638749+0000', 'last_became_active': '2023-10-11T11:15:31.679465+0000', 'last_became_peered': '2023-10-11T11:15:31.679465+0000', 'last_unstale': '2023-10-11T11:19:00.636960+0000', 'last_undegraded': '2023-10-11T11:19:00.636960+0000', 'last_fullsized': '2023-10-11T11:15:31.659135+0000', 'mapping_epoch': 480, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_clean_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-12T16:03:42.656692+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-10-11T11:20:02.398 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-10-11T11:20:02.799 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T11:20:02.799 INFO:teuthology.orchestra.run.smithi111.stdout:{"fsid":"022ce6f2-9a5b-45ad-8b77-a1332d92f228","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2198,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":500,"num_osds":3,"num_up_osds":2,"osd_up_since":1697022930,"num_in_osds":3,"osd_in_since":1697021005,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":2},{"state_name":"active+undersized+degraded","count":1},{"state_name":"active+clean+remapped","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3170594816,"bytes_avail":286739697664,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":36,"modified":"2023-10-11T11:19:04.644408+0000","services":{}},"progress_events":{}} 2023-10-11T11:20:02.814 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-10-11T11:20:02.814 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-10-11T11:20:03.131 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T11:20:03.133 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2023-10-11T11:20:03.144 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":1592,"stamp":"2023-10-11T11:20:02.658169+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":11,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":107,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3096284,"kb_used_data":418996,"kb_used_omap":292,"kb_used_meta":2676955,"kb_avail":280019236,"statfs":{"total":289910292480,"available":286739697664,"internally_reserved":0,"allocated":429051904,"data_stored":420453125,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299441,"internal_metadata":2741202511},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002924"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":517,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-10-11T11:19:02.772828+0000","last_change":"2023-10-11T11:15:31.661747+0000","last_active":"2023-10-11T11:19:02.772828+0000","last_peered":"2023-10-11T11:19:02.772828+0000","last_clean":"2023-10-11T11:15:27.638215+0000","last_became_active":"2023-10-11T11:15:31.661747+0000","last_became_peered":"2023-10-11T11:15:31.661747+0000","last_unstale":"2023-10-11T11:19:02.772828+0000","last_undegraded":"2023-10-11T11:19:02.772828+0000","last_fullsized":"2023-10-11T11:15:31.649336+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:15:25.047039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":29,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-10-11T11:19:02.774213+0000","last_change":"2023-10-11T11:15:31.671363+0000","last_active":"2023-10-11T11:19:02.774213+0000","last_peered":"2023-10-11T11:19:02.774213+0000","last_clean":"2023-10-11T10:43:40.751300+0000","last_became_active":"2023-10-11T11:15:31.671363+0000","last_became_peered":"2023-10-11T11:15:31.671363+0000","last_unstale":"2023-10-11T11:19:02.774213+0000","last_undegraded":"2023-10-11T11:19:02.774213+0000","last_fullsized":"2023-10-11T11:15:31.652001+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:01:53.884964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":512,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-10-11T11:19:03.429640+0000","last_change":"2023-10-11T10:43:41.772781+0000","last_active":"2023-10-11T11:19:03.429640+0000","last_peered":"2023-10-11T11:19:03.429640+0000","last_clean":"2023-10-11T11:19:03.429640+0000","last_became_active":"2023-10-11T10:43:41.772443+0000","last_became_peered":"2023-10-11T10:43:41.772443+0000","last_unstale":"2023-10-11T11:19:03.429640+0000","last_undegraded":"2023-10-11T11:19:03.429640+0000","last_fullsized":"2023-10-11T11:19:03.429640+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T14:15:30.547284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":513,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-10-11T11:19:02.772211+0000","last_change":"2023-10-11T10:43:41.772615+0000","last_active":"2023-10-11T11:19:02.772211+0000","last_peered":"2023-10-11T11:19:02.772211+0000","last_clean":"2023-10-11T11:19:02.772211+0000","last_became_active":"2023-10-11T10:43:41.772290+0000","last_became_peered":"2023-10-11T10:43:41.772290+0000","last_unstale":"2023-10-11T11:19:02.772211+0000","last_undegraded":"2023-10-11T11:19:02.772211+0000","last_fullsized":"2023-10-11T11:19:02.772211+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:26:06.909963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":648,"reported_epoch":500,"state":"active+undersized+degraded","last_fresh":"2023-10-11T11:19:02.769176+0000","last_change":"2023-10-11T11:15:31.654022+0000","last_active":"2023-10-11T11:19:02.769176+0000","last_peered":"2023-10-11T11:19:02.769176+0000","last_clean":"2023-10-11T11:15:27.636884+0000","last_became_active":"2023-10-11T11:15:31.654022+0000","last_became_peered":"2023-10-11T11:15:31.654022+0000","last_unstale":"2023-10-11T11:19:02.769176+0000","last_undegraded":"2023-10-11T11:15:31.646946+0000","last_fullsized":"2023-10-11T11:15:31.646751+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:36.739637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:36.739637+0000","last_clean_scrub_stamp":"2023-10-11T10:43:36.739637+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T11:33:33.696447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":520,"reported_epoch":500,"state":"active+clean+remapped","last_fresh":"2023-10-11T11:19:03.429429+0000","last_change":"2023-10-11T11:15:33.660200+0000","last_active":"2023-10-11T11:19:03.429429+0000","last_peered":"2023-10-11T11:19:03.429429+0000","last_clean":"2023-10-11T11:19:03.429429+0000","last_became_active":"2023-10-11T11:15:33.659619+0000","last_became_peered":"2023-10-11T11:15:33.659619+0000","last_unstale":"2023-10-11T11:19:03.429429+0000","last_undegraded":"2023-10-11T11:19:03.429429+0000","last_fullsized":"2023-10-11T11:19:03.429429+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T17:21:16.866392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":517,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-10-11T11:19:02.775128+0000","last_change":"2023-10-11T11:15:31.670813+0000","last_active":"2023-10-11T11:19:02.775128+0000","last_peered":"2023-10-11T11:19:02.775128+0000","last_clean":"2023-10-11T11:15:27.638219+0000","last_became_active":"2023-10-11T11:15:31.670813+0000","last_became_peered":"2023-10-11T11:15:31.670813+0000","last_unstale":"2023-10-11T11:19:02.775128+0000","last_undegraded":"2023-10-11T11:19:02.775128+0000","last_fullsized":"2023-10-11T11:15:31.658785+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:41:50.872346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":516,"reported_epoch":499,"state":"active+undersized","last_fresh":"2023-10-11T11:19:00.631668+0000","last_change":"2023-10-11T11:15:31.666378+0000","last_active":"2023-10-11T11:19:00.631668+0000","last_peered":"2023-10-11T11:19:00.631668+0000","last_clean":"2023-10-11T11:15:27.638303+0000","last_became_active":"2023-10-11T11:15:31.666378+0000","last_became_peered":"2023-10-11T11:15:31.666378+0000","last_unstale":"2023-10-11T11:19:00.631668+0000","last_undegraded":"2023-10-11T11:19:00.631668+0000","last_fullsized":"2023-10-11T11:15:31.649186+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T15:35:19.331209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":516,"reported_epoch":499,"state":"active+undersized","last_fresh":"2023-10-11T11:19:00.636960+0000","last_change":"2023-10-11T11:15:31.679465+0000","last_active":"2023-10-11T11:19:00.636960+0000","last_peered":"2023-10-11T11:19:00.636960+0000","last_clean":"2023-10-11T11:15:27.638749+0000","last_became_active":"2023-10-11T11:15:31.679465+0000","last_became_peered":"2023-10-11T11:15:31.679465+0000","last_unstale":"2023-10-11T11:19:00.636960+0000","last_undegraded":"2023-10-11T11:19:00.636960+0000","last_fullsized":"2023-10-11T11:15:31.659135+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T16:03:42.656692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":11,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738808,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":785348,"kb_used_data":5812,"kb_used_omap":96,"kb_used_meta":779423,"kb_avail":93586492,"statfs":{"total":96636764160,"available":95832567808,"internally_reserved":0,"allocated":5951488,"data_stored":3106918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":98609,"internal_metadata":798129871},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 11:14:37 2023","interfaces":[{"interface":"back","average":{"1min":0.314,"5min":0.316,"15min":0.443},"min":{"1min":0.24399999999999999,"5min":0.14799999999999999,"15min":0.14599999999999999},"max":{"1min":0.48499999999999999,"5min":1.712,"15min":9.6400000000000006},"last":0.31},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.35499999999999998,"15min":0.41299999999999998},"min":{"1min":0.20899999999999999,"5min":0.17199999999999999,"15min":0.16400000000000001},"max":{"1min":3.8180000000000001,"5min":3.8180000000000001,"15min":6.3200000000000003},"last":0.26500000000000001}]},{"osd":1,"last update":"Wed Oct 11 11:19:40 2023","interfaces":[{"interface":"back","average":{"1min":0.29599999999999999,"5min":0.28499999999999998,"15min":0.32300000000000001},"min":{"1min":0.17699999999999999,"5min":0.154,"15min":0.154},"max":{"1min":0.40600000000000003,"5min":0.41999999999999998,"15min":2.9540000000000002},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.317,"5min":0.31,"15min":0.36199999999999999},"min":{"1min":0.20899999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.45000000000000001,"5min":0.49099999999999999,"15min":3.2719999999999998},"last":0.42399999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738756,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1484496,"kb_used_data":406792,"kb_used_omap":107,"kb_used_meta":1077588,"kb_avail":92887344,"statfs":{"total":96636764160,"available":95116640256,"internally_reserved":0,"allocated":416555008,"data_stored":413648921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110309,"internal_metadata":1103450395},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Oct 11 11:14:30 2023","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.35399999999999998,"15min":0.40600000000000003},"min":{"1min":0.18099999999999999,"5min":0.155,"15min":0.14899999999999999},"max":{"1min":1.4319999999999999,"5min":2.3460000000000001,"15min":6.593},"last":0.26200000000000001},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.39100000000000001,"15min":0.48199999999999998},"min":{"1min":0.189,"5min":0.189,"15min":0.183},"max":{"1min":2.2120000000000002,"5min":2.4089999999999998,"15min":6.5960000000000001},"last":0.316}]},{"osd":2,"last update":"Wed Oct 11 11:14:30 2023","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.35499999999999998,"15min":0.47099999999999997},"min":{"1min":0.20300000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":1.2490000000000001,"5min":2.665,"15min":8.4730000000000008},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.375,"15min":0.442},"min":{"1min":0.23200000000000001,"5min":0.20499999999999999,"15min":0.19600000000000001},"max":{"1min":1.28,"5min":2.6230000000000002,"15min":7.0129999999999999},"last":0.33600000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738809,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":826440,"kb_used_data":6392,"kb_used_omap":88,"kb_used_meta":819943,"kb_avail":93545400,"statfs":{"total":96636764160,"available":95790489600,"internally_reserved":0,"allocated":6545408,"data_stored":3697286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":90523,"internal_metadata":839622245},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 11:14:33 2023","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.42799999999999999,"15min":0.41099999999999998},"min":{"1min":0.151,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":1.2809999999999999,"5min":3.5739999999999998,"15min":8.2479999999999993},"last":0.223},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.48299999999999998,"15min":0.49199999999999999},"min":{"1min":0.20699999999999999,"5min":0.182,"15min":0.17999999999999999},"max":{"1min":1.323,"5min":3.6080000000000001,"15min":8.2870000000000008},"last":0.28399999999999997}]},{"osd":2,"last update":"Wed Oct 11 11:19:41 2023","interfaces":[{"interface":"back","average":{"1min":0.314,"5min":0.35399999999999998,"15min":0.35899999999999999},"min":{"1min":0.186,"5min":0.14699999999999999,"15min":0.14699999999999999},"max":{"1min":0.45500000000000002,"5min":7.4829999999999997,"15min":10.617000000000001},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.39400000000000002,"15min":0.35099999999999998},"min":{"1min":0.21199999999999999,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.49099999999999999,"5min":7.5350000000000001,"15min":7.5350000000000001},"last":0.307}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-11T11:20:06.146 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-10-11T11:20:06.474 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T11:20:06.476 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2023-10-11T11:20:06.488 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":1593,"stamp":"2023-10-11T11:20:04.658752+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":11,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":107,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3096284,"kb_used_data":418996,"kb_used_omap":292,"kb_used_meta":2676955,"kb_avail":280019236,"statfs":{"total":289910292480,"available":286739697664,"internally_reserved":0,"allocated":429051904,"data_stored":420453125,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299441,"internal_metadata":2741202511},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002912"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":517,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-10-11T11:19:02.772828+0000","last_change":"2023-10-11T11:15:31.661747+0000","last_active":"2023-10-11T11:19:02.772828+0000","last_peered":"2023-10-11T11:19:02.772828+0000","last_clean":"2023-10-11T11:15:27.638215+0000","last_became_active":"2023-10-11T11:15:31.661747+0000","last_became_peered":"2023-10-11T11:15:31.661747+0000","last_unstale":"2023-10-11T11:19:02.772828+0000","last_undegraded":"2023-10-11T11:19:02.772828+0000","last_fullsized":"2023-10-11T11:15:31.649336+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:15:25.047039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":29,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-10-11T11:19:02.774213+0000","last_change":"2023-10-11T11:15:31.671363+0000","last_active":"2023-10-11T11:19:02.774213+0000","last_peered":"2023-10-11T11:19:02.774213+0000","last_clean":"2023-10-11T10:43:40.751300+0000","last_became_active":"2023-10-11T11:15:31.671363+0000","last_became_peered":"2023-10-11T11:15:31.671363+0000","last_unstale":"2023-10-11T11:19:02.774213+0000","last_undegraded":"2023-10-11T11:19:02.774213+0000","last_fullsized":"2023-10-11T11:15:31.652001+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:01:53.884964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":512,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-10-11T11:19:03.429640+0000","last_change":"2023-10-11T10:43:41.772781+0000","last_active":"2023-10-11T11:19:03.429640+0000","last_peered":"2023-10-11T11:19:03.429640+0000","last_clean":"2023-10-11T11:19:03.429640+0000","last_became_active":"2023-10-11T10:43:41.772443+0000","last_became_peered":"2023-10-11T10:43:41.772443+0000","last_unstale":"2023-10-11T11:19:03.429640+0000","last_undegraded":"2023-10-11T11:19:03.429640+0000","last_fullsized":"2023-10-11T11:19:03.429640+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T14:15:30.547284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":513,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-10-11T11:19:02.772211+0000","last_change":"2023-10-11T10:43:41.772615+0000","last_active":"2023-10-11T11:19:02.772211+0000","last_peered":"2023-10-11T11:19:02.772211+0000","last_clean":"2023-10-11T11:19:02.772211+0000","last_became_active":"2023-10-11T10:43:41.772290+0000","last_became_peered":"2023-10-11T10:43:41.772290+0000","last_unstale":"2023-10-11T11:19:02.772211+0000","last_undegraded":"2023-10-11T11:19:02.772211+0000","last_fullsized":"2023-10-11T11:19:02.772211+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:26:06.909963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":648,"reported_epoch":500,"state":"active+undersized+degraded","last_fresh":"2023-10-11T11:19:02.769176+0000","last_change":"2023-10-11T11:15:31.654022+0000","last_active":"2023-10-11T11:19:02.769176+0000","last_peered":"2023-10-11T11:19:02.769176+0000","last_clean":"2023-10-11T11:15:27.636884+0000","last_became_active":"2023-10-11T11:15:31.654022+0000","last_became_peered":"2023-10-11T11:15:31.654022+0000","last_unstale":"2023-10-11T11:19:02.769176+0000","last_undegraded":"2023-10-11T11:15:31.646946+0000","last_fullsized":"2023-10-11T11:15:31.646751+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:36.739637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:36.739637+0000","last_clean_scrub_stamp":"2023-10-11T10:43:36.739637+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T11:33:33.696447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":520,"reported_epoch":500,"state":"active+clean+remapped","last_fresh":"2023-10-11T11:19:03.429429+0000","last_change":"2023-10-11T11:15:33.660200+0000","last_active":"2023-10-11T11:19:03.429429+0000","last_peered":"2023-10-11T11:19:03.429429+0000","last_clean":"2023-10-11T11:19:03.429429+0000","last_became_active":"2023-10-11T11:15:33.659619+0000","last_became_peered":"2023-10-11T11:15:33.659619+0000","last_unstale":"2023-10-11T11:19:03.429429+0000","last_undegraded":"2023-10-11T11:19:03.429429+0000","last_fullsized":"2023-10-11T11:19:03.429429+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T17:21:16.866392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":517,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-10-11T11:19:02.775128+0000","last_change":"2023-10-11T11:15:31.670813+0000","last_active":"2023-10-11T11:19:02.775128+0000","last_peered":"2023-10-11T11:19:02.775128+0000","last_clean":"2023-10-11T11:15:27.638219+0000","last_became_active":"2023-10-11T11:15:31.670813+0000","last_became_peered":"2023-10-11T11:15:31.670813+0000","last_unstale":"2023-10-11T11:19:02.775128+0000","last_undegraded":"2023-10-11T11:19:02.775128+0000","last_fullsized":"2023-10-11T11:15:31.658785+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:41:50.872346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":516,"reported_epoch":499,"state":"active+undersized","last_fresh":"2023-10-11T11:19:00.631668+0000","last_change":"2023-10-11T11:15:31.666378+0000","last_active":"2023-10-11T11:19:00.631668+0000","last_peered":"2023-10-11T11:19:00.631668+0000","last_clean":"2023-10-11T11:15:27.638303+0000","last_became_active":"2023-10-11T11:15:31.666378+0000","last_became_peered":"2023-10-11T11:15:31.666378+0000","last_unstale":"2023-10-11T11:19:00.631668+0000","last_undegraded":"2023-10-11T11:19:00.631668+0000","last_fullsized":"2023-10-11T11:15:31.649186+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T15:35:19.331209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":516,"reported_epoch":499,"state":"active+undersized","last_fresh":"2023-10-11T11:19:00.636960+0000","last_change":"2023-10-11T11:15:31.679465+0000","last_active":"2023-10-11T11:19:00.636960+0000","last_peered":"2023-10-11T11:19:00.636960+0000","last_clean":"2023-10-11T11:15:27.638749+0000","last_became_active":"2023-10-11T11:15:31.679465+0000","last_became_peered":"2023-10-11T11:15:31.679465+0000","last_unstale":"2023-10-11T11:19:00.636960+0000","last_undegraded":"2023-10-11T11:19:00.636960+0000","last_fullsized":"2023-10-11T11:15:31.659135+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T16:03:42.656692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":11,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738808,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":785348,"kb_used_data":5812,"kb_used_omap":96,"kb_used_meta":779423,"kb_avail":93586492,"statfs":{"total":96636764160,"available":95832567808,"internally_reserved":0,"allocated":5951488,"data_stored":3106918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":98609,"internal_metadata":798129871},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 11:14:37 2023","interfaces":[{"interface":"back","average":{"1min":0.314,"5min":0.316,"15min":0.443},"min":{"1min":0.24399999999999999,"5min":0.14799999999999999,"15min":0.14599999999999999},"max":{"1min":0.48499999999999999,"5min":1.712,"15min":9.6400000000000006},"last":0.31},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.35499999999999998,"15min":0.41299999999999998},"min":{"1min":0.20899999999999999,"5min":0.17199999999999999,"15min":0.16400000000000001},"max":{"1min":3.8180000000000001,"5min":3.8180000000000001,"15min":6.3200000000000003},"last":0.26500000000000001}]},{"osd":1,"last update":"Wed Oct 11 11:19:40 2023","interfaces":[{"interface":"back","average":{"1min":0.29599999999999999,"5min":0.28499999999999998,"15min":0.32300000000000001},"min":{"1min":0.17699999999999999,"5min":0.154,"15min":0.154},"max":{"1min":0.40600000000000003,"5min":0.41999999999999998,"15min":2.9540000000000002},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.317,"5min":0.31,"15min":0.36199999999999999},"min":{"1min":0.20899999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.45000000000000001,"5min":0.49099999999999999,"15min":3.2719999999999998},"last":0.42399999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738756,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1484496,"kb_used_data":406792,"kb_used_omap":107,"kb_used_meta":1077588,"kb_avail":92887344,"statfs":{"total":96636764160,"available":95116640256,"internally_reserved":0,"allocated":416555008,"data_stored":413648921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110309,"internal_metadata":1103450395},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Oct 11 11:14:30 2023","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.35399999999999998,"15min":0.40600000000000003},"min":{"1min":0.18099999999999999,"5min":0.155,"15min":0.14899999999999999},"max":{"1min":1.4319999999999999,"5min":2.3460000000000001,"15min":6.593},"last":0.26200000000000001},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.39100000000000001,"15min":0.48199999999999998},"min":{"1min":0.189,"5min":0.189,"15min":0.183},"max":{"1min":2.2120000000000002,"5min":2.4089999999999998,"15min":6.5960000000000001},"last":0.316}]},{"osd":2,"last update":"Wed Oct 11 11:14:30 2023","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.35499999999999998,"15min":0.47099999999999997},"min":{"1min":0.20300000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":1.2490000000000001,"5min":2.665,"15min":8.4730000000000008},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.375,"15min":0.442},"min":{"1min":0.23200000000000001,"5min":0.20499999999999999,"15min":0.19600000000000001},"max":{"1min":1.28,"5min":2.6230000000000002,"15min":7.0129999999999999},"last":0.33600000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738809,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":826440,"kb_used_data":6392,"kb_used_omap":88,"kb_used_meta":819943,"kb_avail":93545400,"statfs":{"total":96636764160,"available":95790489600,"internally_reserved":0,"allocated":6545408,"data_stored":3697286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":90523,"internal_metadata":839622245},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 11:14:33 2023","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.42799999999999999,"15min":0.41099999999999998},"min":{"1min":0.151,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":1.2809999999999999,"5min":3.5739999999999998,"15min":8.2479999999999993},"last":0.223},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.48299999999999998,"15min":0.49199999999999999},"min":{"1min":0.20699999999999999,"5min":0.182,"15min":0.17999999999999999},"max":{"1min":1.323,"5min":3.6080000000000001,"15min":8.2870000000000008},"last":0.28399999999999997}]},{"osd":2,"last update":"Wed Oct 11 11:19:41 2023","interfaces":[{"interface":"back","average":{"1min":0.314,"5min":0.35399999999999998,"15min":0.35899999999999999},"min":{"1min":0.186,"5min":0.14699999999999999,"15min":0.14699999999999999},"max":{"1min":0.45500000000000002,"5min":7.4829999999999997,"15min":10.617000000000001},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.39400000000000002,"15min":0.35099999999999998},"min":{"1min":0.21199999999999999,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.49099999999999999,"5min":7.5350000000000001,"15min":7.5350000000000001},"last":0.307}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-11T11:20:06.489 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-10-11T11:20:06.805 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T11:20:06.808 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2023-10-11T11:20:06.819 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":1594,"stamp":"2023-10-11T11:20:06.659371+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":11,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":107,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3096284,"kb_used_data":418996,"kb_used_omap":292,"kb_used_meta":2676955,"kb_avail":280019236,"statfs":{"total":289910292480,"available":286739697664,"internally_reserved":0,"allocated":429051904,"data_stored":420453125,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299441,"internal_metadata":2741202511},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003144"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":517,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-10-11T11:19:02.772828+0000","last_change":"2023-10-11T11:15:31.661747+0000","last_active":"2023-10-11T11:19:02.772828+0000","last_peered":"2023-10-11T11:19:02.772828+0000","last_clean":"2023-10-11T11:15:27.638215+0000","last_became_active":"2023-10-11T11:15:31.661747+0000","last_became_peered":"2023-10-11T11:15:31.661747+0000","last_unstale":"2023-10-11T11:19:02.772828+0000","last_undegraded":"2023-10-11T11:19:02.772828+0000","last_fullsized":"2023-10-11T11:15:31.649336+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:15:25.047039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":29,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-10-11T11:19:02.774213+0000","last_change":"2023-10-11T11:15:31.671363+0000","last_active":"2023-10-11T11:19:02.774213+0000","last_peered":"2023-10-11T11:19:02.774213+0000","last_clean":"2023-10-11T10:43:40.751300+0000","last_became_active":"2023-10-11T11:15:31.671363+0000","last_became_peered":"2023-10-11T11:15:31.671363+0000","last_unstale":"2023-10-11T11:19:02.774213+0000","last_undegraded":"2023-10-11T11:19:02.774213+0000","last_fullsized":"2023-10-11T11:15:31.652001+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:01:53.884964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":512,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-10-11T11:19:03.429640+0000","last_change":"2023-10-11T10:43:41.772781+0000","last_active":"2023-10-11T11:19:03.429640+0000","last_peered":"2023-10-11T11:19:03.429640+0000","last_clean":"2023-10-11T11:19:03.429640+0000","last_became_active":"2023-10-11T10:43:41.772443+0000","last_became_peered":"2023-10-11T10:43:41.772443+0000","last_unstale":"2023-10-11T11:19:03.429640+0000","last_undegraded":"2023-10-11T11:19:03.429640+0000","last_fullsized":"2023-10-11T11:19:03.429640+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T14:15:30.547284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":513,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-10-11T11:19:02.772211+0000","last_change":"2023-10-11T10:43:41.772615+0000","last_active":"2023-10-11T11:19:02.772211+0000","last_peered":"2023-10-11T11:19:02.772211+0000","last_clean":"2023-10-11T11:19:02.772211+0000","last_became_active":"2023-10-11T10:43:41.772290+0000","last_became_peered":"2023-10-11T10:43:41.772290+0000","last_unstale":"2023-10-11T11:19:02.772211+0000","last_undegraded":"2023-10-11T11:19:02.772211+0000","last_fullsized":"2023-10-11T11:19:02.772211+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:26:06.909963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":648,"reported_epoch":500,"state":"active+undersized+degraded","last_fresh":"2023-10-11T11:19:02.769176+0000","last_change":"2023-10-11T11:15:31.654022+0000","last_active":"2023-10-11T11:19:02.769176+0000","last_peered":"2023-10-11T11:19:02.769176+0000","last_clean":"2023-10-11T11:15:27.636884+0000","last_became_active":"2023-10-11T11:15:31.654022+0000","last_became_peered":"2023-10-11T11:15:31.654022+0000","last_unstale":"2023-10-11T11:19:02.769176+0000","last_undegraded":"2023-10-11T11:15:31.646946+0000","last_fullsized":"2023-10-11T11:15:31.646751+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:36.739637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:36.739637+0000","last_clean_scrub_stamp":"2023-10-11T10:43:36.739637+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T11:33:33.696447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":520,"reported_epoch":500,"state":"active+clean+remapped","last_fresh":"2023-10-11T11:19:03.429429+0000","last_change":"2023-10-11T11:15:33.660200+0000","last_active":"2023-10-11T11:19:03.429429+0000","last_peered":"2023-10-11T11:19:03.429429+0000","last_clean":"2023-10-11T11:19:03.429429+0000","last_became_active":"2023-10-11T11:15:33.659619+0000","last_became_peered":"2023-10-11T11:15:33.659619+0000","last_unstale":"2023-10-11T11:19:03.429429+0000","last_undegraded":"2023-10-11T11:19:03.429429+0000","last_fullsized":"2023-10-11T11:19:03.429429+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T17:21:16.866392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":517,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-10-11T11:19:02.775128+0000","last_change":"2023-10-11T11:15:31.670813+0000","last_active":"2023-10-11T11:19:02.775128+0000","last_peered":"2023-10-11T11:19:02.775128+0000","last_clean":"2023-10-11T11:15:27.638219+0000","last_became_active":"2023-10-11T11:15:31.670813+0000","last_became_peered":"2023-10-11T11:15:31.670813+0000","last_unstale":"2023-10-11T11:19:02.775128+0000","last_undegraded":"2023-10-11T11:19:02.775128+0000","last_fullsized":"2023-10-11T11:15:31.658785+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:41:50.872346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":516,"reported_epoch":499,"state":"active+undersized","last_fresh":"2023-10-11T11:19:00.631668+0000","last_change":"2023-10-11T11:15:31.666378+0000","last_active":"2023-10-11T11:19:00.631668+0000","last_peered":"2023-10-11T11:19:00.631668+0000","last_clean":"2023-10-11T11:15:27.638303+0000","last_became_active":"2023-10-11T11:15:31.666378+0000","last_became_peered":"2023-10-11T11:15:31.666378+0000","last_unstale":"2023-10-11T11:19:00.631668+0000","last_undegraded":"2023-10-11T11:19:00.631668+0000","last_fullsized":"2023-10-11T11:15:31.649186+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T15:35:19.331209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":516,"reported_epoch":499,"state":"active+undersized","last_fresh":"2023-10-11T11:19:00.636960+0000","last_change":"2023-10-11T11:15:31.679465+0000","last_active":"2023-10-11T11:19:00.636960+0000","last_peered":"2023-10-11T11:19:00.636960+0000","last_clean":"2023-10-11T11:15:27.638749+0000","last_became_active":"2023-10-11T11:15:31.679465+0000","last_became_peered":"2023-10-11T11:15:31.679465+0000","last_unstale":"2023-10-11T11:19:00.636960+0000","last_undegraded":"2023-10-11T11:19:00.636960+0000","last_fullsized":"2023-10-11T11:15:31.659135+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T16:03:42.656692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":11,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738809,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":785348,"kb_used_data":5812,"kb_used_omap":96,"kb_used_meta":779423,"kb_avail":93586492,"statfs":{"total":96636764160,"available":95832567808,"internally_reserved":0,"allocated":5951488,"data_stored":3106918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":98609,"internal_metadata":798129871},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 11:14:37 2023","interfaces":[{"interface":"back","average":{"1min":0.314,"5min":0.316,"15min":0.443},"min":{"1min":0.24399999999999999,"5min":0.14799999999999999,"15min":0.14599999999999999},"max":{"1min":0.48499999999999999,"5min":1.712,"15min":9.6400000000000006},"last":0.31},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.35499999999999998,"15min":0.41299999999999998},"min":{"1min":0.20899999999999999,"5min":0.17199999999999999,"15min":0.16400000000000001},"max":{"1min":3.8180000000000001,"5min":3.8180000000000001,"15min":6.3200000000000003},"last":0.26500000000000001}]},{"osd":1,"last update":"Wed Oct 11 11:19:40 2023","interfaces":[{"interface":"back","average":{"1min":0.29599999999999999,"5min":0.28499999999999998,"15min":0.32300000000000001},"min":{"1min":0.17699999999999999,"5min":0.154,"15min":0.154},"max":{"1min":0.40600000000000003,"5min":0.41999999999999998,"15min":2.9540000000000002},"last":0.312},{"interface":"front","average":{"1min":0.317,"5min":0.31,"15min":0.36199999999999999},"min":{"1min":0.20899999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.45000000000000001,"5min":0.49099999999999999,"15min":3.2719999999999998},"last":0.35099999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738756,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1484496,"kb_used_data":406792,"kb_used_omap":107,"kb_used_meta":1077588,"kb_avail":92887344,"statfs":{"total":96636764160,"available":95116640256,"internally_reserved":0,"allocated":416555008,"data_stored":413648921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110309,"internal_metadata":1103450395},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Oct 11 11:14:30 2023","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.35399999999999998,"15min":0.40600000000000003},"min":{"1min":0.18099999999999999,"5min":0.155,"15min":0.14899999999999999},"max":{"1min":1.4319999999999999,"5min":2.3460000000000001,"15min":6.593},"last":0.26200000000000001},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.39100000000000001,"15min":0.48199999999999998},"min":{"1min":0.189,"5min":0.189,"15min":0.183},"max":{"1min":2.2120000000000002,"5min":2.4089999999999998,"15min":6.5960000000000001},"last":0.316}]},{"osd":2,"last update":"Wed Oct 11 11:14:30 2023","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.35499999999999998,"15min":0.47099999999999997},"min":{"1min":0.20300000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":1.2490000000000001,"5min":2.665,"15min":8.4730000000000008},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.375,"15min":0.442},"min":{"1min":0.23200000000000001,"5min":0.20499999999999999,"15min":0.19600000000000001},"max":{"1min":1.28,"5min":2.6230000000000002,"15min":7.0129999999999999},"last":0.33600000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738810,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":826440,"kb_used_data":6392,"kb_used_omap":88,"kb_used_meta":819943,"kb_avail":93545400,"statfs":{"total":96636764160,"available":95790489600,"internally_reserved":0,"allocated":6545408,"data_stored":3697286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":90523,"internal_metadata":839622245},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 11:14:33 2023","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.42799999999999999,"15min":0.41099999999999998},"min":{"1min":0.151,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":1.2809999999999999,"5min":3.5739999999999998,"15min":8.2479999999999993},"last":0.223},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.48299999999999998,"15min":0.49199999999999999},"min":{"1min":0.20699999999999999,"5min":0.182,"15min":0.17999999999999999},"max":{"1min":1.323,"5min":3.6080000000000001,"15min":8.2870000000000008},"last":0.28399999999999997}]},{"osd":2,"last update":"Wed Oct 11 11:19:41 2023","interfaces":[{"interface":"back","average":{"1min":0.314,"5min":0.35399999999999998,"15min":0.35899999999999999},"min":{"1min":0.186,"5min":0.14699999999999999,"15min":0.14699999999999999},"max":{"1min":0.45500000000000002,"5min":7.4829999999999997,"15min":10.617000000000001},"last":0.33500000000000002},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.39400000000000002,"15min":0.35099999999999998},"min":{"1min":0.21199999999999999,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.49099999999999999,"5min":7.5350000000000001,"15min":7.5350000000000001},"last":0.36899999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-11T11:20:06.820 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-10-11T11:20:06.820 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 517, 'reported_epoch': 500, 'state': 'active+undersized', 'last_fresh': '2023-10-11T11:19:02.772828+0000', 'last_change': '2023-10-11T11:15:31.661747+0000', 'last_active': '2023-10-11T11:19:02.772828+0000', 'last_peered': '2023-10-11T11:19:02.772828+0000', 'last_clean': '2023-10-11T11:15:27.638215+0000', 'last_became_active': '2023-10-11T11:15:31.661747+0000', 'last_became_peered': '2023-10-11T11:15:31.661747+0000', 'last_unstale': '2023-10-11T11:19:02.772828+0000', 'last_undegraded': '2023-10-11T11:19:02.772828+0000', 'last_fullsized': '2023-10-11T11:15:31.649336+0000', 'mapping_epoch': 480, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_clean_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-12T22:15:25.047039+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-10-11T11:20:06.821 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-10-11T11:20:06.821 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 29, 'reported_epoch': 500, 'state': 'active+undersized', 'last_fresh': '2023-10-11T11:19:02.774213+0000', 'last_change': '2023-10-11T11:15:31.671363+0000', 'last_active': '2023-10-11T11:19:02.774213+0000', 'last_peered': '2023-10-11T11:19:02.774213+0000', 'last_clean': '2023-10-11T10:43:40.751300+0000', 'last_became_active': '2023-10-11T11:15:31.671363+0000', 'last_became_peered': '2023-10-11T11:15:31.671363+0000', 'last_unstale': '2023-10-11T11:19:02.774213+0000', 'last_undegraded': '2023-10-11T11:19:02.774213+0000', 'last_fullsized': '2023-10-11T11:15:31.652001+0000', 'mapping_epoch': 480, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_clean_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-12T21:01:53.884964+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-10-11T11:20:06.821 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-10-11T11:20:06.822 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "11'97", 'reported_seq': 648, 'reported_epoch': 500, 'state': 'active+undersized+degraded', 'last_fresh': '2023-10-11T11:19:02.769176+0000', 'last_change': '2023-10-11T11:15:31.654022+0000', 'last_active': '2023-10-11T11:19:02.769176+0000', 'last_peered': '2023-10-11T11:19:02.769176+0000', 'last_clean': '2023-10-11T11:15:27.636884+0000', 'last_became_active': '2023-10-11T11:15:31.654022+0000', 'last_became_peered': '2023-10-11T11:15:31.654022+0000', 'last_unstale': '2023-10-11T11:19:02.769176+0000', 'last_undegraded': '2023-10-11T11:15:31.646946+0000', 'last_fullsized': '2023-10-11T11:15:31.646751+0000', 'mapping_epoch': 480, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-11T10:43:36.739637+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-11T10:43:36.739637+0000', 'last_clean_scrub_stamp': '2023-10-11T10:43:36.739637+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-12T11:33:33.696447+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-10-11T11:20:06.822 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.0 is not active+clean 2023-10-11T11:20:06.822 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 520, 'reported_epoch': 500, 'state': 'active+clean+remapped', 'last_fresh': '2023-10-11T11:19:03.429429+0000', 'last_change': '2023-10-11T11:15:33.660200+0000', 'last_active': '2023-10-11T11:19:03.429429+0000', 'last_peered': '2023-10-11T11:19:03.429429+0000', 'last_clean': '2023-10-11T11:19:03.429429+0000', 'last_became_active': '2023-10-11T11:15:33.659619+0000', 'last_became_peered': '2023-10-11T11:15:33.659619+0000', 'last_unstale': '2023-10-11T11:19:03.429429+0000', 'last_undegraded': '2023-10-11T11:19:03.429429+0000', 'last_fullsized': '2023-10-11T11:19:03.429429+0000', 'mapping_epoch': 482, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 483, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_clean_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-12T17:21:16.866392+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2, 1], 'avail_no_missing': ['2', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-10-11T11:20:06.823 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-10-11T11:20:06.823 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 517, 'reported_epoch': 500, 'state': 'active+undersized', 'last_fresh': '2023-10-11T11:19:02.775128+0000', 'last_change': '2023-10-11T11:15:31.670813+0000', 'last_active': '2023-10-11T11:19:02.775128+0000', 'last_peered': '2023-10-11T11:19:02.775128+0000', 'last_clean': '2023-10-11T11:15:27.638219+0000', 'last_became_active': '2023-10-11T11:15:31.670813+0000', 'last_became_peered': '2023-10-11T11:15:31.670813+0000', 'last_unstale': '2023-10-11T11:19:02.775128+0000', 'last_undegraded': '2023-10-11T11:19:02.775128+0000', 'last_fullsized': '2023-10-11T11:15:31.658785+0000', 'mapping_epoch': 480, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_clean_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-12T21:41:50.872346+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-10-11T11:20:06.823 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-10-11T11:20:06.824 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 516, 'reported_epoch': 499, 'state': 'active+undersized', 'last_fresh': '2023-10-11T11:19:00.631668+0000', 'last_change': '2023-10-11T11:15:31.666378+0000', 'last_active': '2023-10-11T11:19:00.631668+0000', 'last_peered': '2023-10-11T11:19:00.631668+0000', 'last_clean': '2023-10-11T11:15:27.638303+0000', 'last_became_active': '2023-10-11T11:15:31.666378+0000', 'last_became_peered': '2023-10-11T11:15:31.666378+0000', 'last_unstale': '2023-10-11T11:19:00.631668+0000', 'last_undegraded': '2023-10-11T11:19:00.631668+0000', 'last_fullsized': '2023-10-11T11:15:31.649186+0000', 'mapping_epoch': 480, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_clean_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-12T15:35:19.331209+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-10-11T11:20:06.824 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-10-11T11:20:06.824 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 516, 'reported_epoch': 499, 'state': 'active+undersized', 'last_fresh': '2023-10-11T11:19:00.636960+0000', 'last_change': '2023-10-11T11:15:31.679465+0000', 'last_active': '2023-10-11T11:19:00.636960+0000', 'last_peered': '2023-10-11T11:19:00.636960+0000', 'last_clean': '2023-10-11T11:15:27.638749+0000', 'last_became_active': '2023-10-11T11:15:31.679465+0000', 'last_became_peered': '2023-10-11T11:15:31.679465+0000', 'last_unstale': '2023-10-11T11:19:00.636960+0000', 'last_undegraded': '2023-10-11T11:19:00.636960+0000', 'last_fullsized': '2023-10-11T11:15:31.659135+0000', 'mapping_epoch': 480, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_clean_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-12T16:03:42.656692+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-10-11T11:20:06.825 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-10-11T11:20:07.220 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T11:20:07.220 INFO:teuthology.orchestra.run.smithi111.stdout:{"fsid":"022ce6f2-9a5b-45ad-8b77-a1332d92f228","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2202,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":500,"num_osds":3,"num_up_osds":2,"osd_up_since":1697022930,"num_in_osds":3,"osd_in_since":1697021005,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":2},{"state_name":"active+undersized+degraded","count":1},{"state_name":"active+clean+remapped","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3170594816,"bytes_avail":286739697664,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":36,"modified":"2023-10-11T11:19:04.644408+0000","services":{}},"progress_events":{}} 2023-10-11T11:20:07.234 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-10-11T11:20:07.234 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-10-11T11:20:07.549 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T11:20:07.551 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2023-10-11T11:20:07.563 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":1594,"stamp":"2023-10-11T11:20:06.659371+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":11,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":107,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3096284,"kb_used_data":418996,"kb_used_omap":292,"kb_used_meta":2676955,"kb_avail":280019236,"statfs":{"total":289910292480,"available":286739697664,"internally_reserved":0,"allocated":429051904,"data_stored":420453125,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299441,"internal_metadata":2741202511},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003144"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":517,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-10-11T11:19:02.772828+0000","last_change":"2023-10-11T11:15:31.661747+0000","last_active":"2023-10-11T11:19:02.772828+0000","last_peered":"2023-10-11T11:19:02.772828+0000","last_clean":"2023-10-11T11:15:27.638215+0000","last_became_active":"2023-10-11T11:15:31.661747+0000","last_became_peered":"2023-10-11T11:15:31.661747+0000","last_unstale":"2023-10-11T11:19:02.772828+0000","last_undegraded":"2023-10-11T11:19:02.772828+0000","last_fullsized":"2023-10-11T11:15:31.649336+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:15:25.047039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":29,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-10-11T11:19:02.774213+0000","last_change":"2023-10-11T11:15:31.671363+0000","last_active":"2023-10-11T11:19:02.774213+0000","last_peered":"2023-10-11T11:19:02.774213+0000","last_clean":"2023-10-11T10:43:40.751300+0000","last_became_active":"2023-10-11T11:15:31.671363+0000","last_became_peered":"2023-10-11T11:15:31.671363+0000","last_unstale":"2023-10-11T11:19:02.774213+0000","last_undegraded":"2023-10-11T11:19:02.774213+0000","last_fullsized":"2023-10-11T11:15:31.652001+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:01:53.884964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":512,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-10-11T11:19:03.429640+0000","last_change":"2023-10-11T10:43:41.772781+0000","last_active":"2023-10-11T11:19:03.429640+0000","last_peered":"2023-10-11T11:19:03.429640+0000","last_clean":"2023-10-11T11:19:03.429640+0000","last_became_active":"2023-10-11T10:43:41.772443+0000","last_became_peered":"2023-10-11T10:43:41.772443+0000","last_unstale":"2023-10-11T11:19:03.429640+0000","last_undegraded":"2023-10-11T11:19:03.429640+0000","last_fullsized":"2023-10-11T11:19:03.429640+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T14:15:30.547284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":513,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-10-11T11:19:02.772211+0000","last_change":"2023-10-11T10:43:41.772615+0000","last_active":"2023-10-11T11:19:02.772211+0000","last_peered":"2023-10-11T11:19:02.772211+0000","last_clean":"2023-10-11T11:19:02.772211+0000","last_became_active":"2023-10-11T10:43:41.772290+0000","last_became_peered":"2023-10-11T10:43:41.772290+0000","last_unstale":"2023-10-11T11:19:02.772211+0000","last_undegraded":"2023-10-11T11:19:02.772211+0000","last_fullsized":"2023-10-11T11:19:02.772211+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:26:06.909963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":648,"reported_epoch":500,"state":"active+undersized+degraded","last_fresh":"2023-10-11T11:19:02.769176+0000","last_change":"2023-10-11T11:15:31.654022+0000","last_active":"2023-10-11T11:19:02.769176+0000","last_peered":"2023-10-11T11:19:02.769176+0000","last_clean":"2023-10-11T11:15:27.636884+0000","last_became_active":"2023-10-11T11:15:31.654022+0000","last_became_peered":"2023-10-11T11:15:31.654022+0000","last_unstale":"2023-10-11T11:19:02.769176+0000","last_undegraded":"2023-10-11T11:15:31.646946+0000","last_fullsized":"2023-10-11T11:15:31.646751+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:36.739637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:36.739637+0000","last_clean_scrub_stamp":"2023-10-11T10:43:36.739637+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T11:33:33.696447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":520,"reported_epoch":500,"state":"active+clean+remapped","last_fresh":"2023-10-11T11:19:03.429429+0000","last_change":"2023-10-11T11:15:33.660200+0000","last_active":"2023-10-11T11:19:03.429429+0000","last_peered":"2023-10-11T11:19:03.429429+0000","last_clean":"2023-10-11T11:19:03.429429+0000","last_became_active":"2023-10-11T11:15:33.659619+0000","last_became_peered":"2023-10-11T11:15:33.659619+0000","last_unstale":"2023-10-11T11:19:03.429429+0000","last_undegraded":"2023-10-11T11:19:03.429429+0000","last_fullsized":"2023-10-11T11:19:03.429429+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T17:21:16.866392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":517,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-10-11T11:19:02.775128+0000","last_change":"2023-10-11T11:15:31.670813+0000","last_active":"2023-10-11T11:19:02.775128+0000","last_peered":"2023-10-11T11:19:02.775128+0000","last_clean":"2023-10-11T11:15:27.638219+0000","last_became_active":"2023-10-11T11:15:31.670813+0000","last_became_peered":"2023-10-11T11:15:31.670813+0000","last_unstale":"2023-10-11T11:19:02.775128+0000","last_undegraded":"2023-10-11T11:19:02.775128+0000","last_fullsized":"2023-10-11T11:15:31.658785+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:41:50.872346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":516,"reported_epoch":499,"state":"active+undersized","last_fresh":"2023-10-11T11:19:00.631668+0000","last_change":"2023-10-11T11:15:31.666378+0000","last_active":"2023-10-11T11:19:00.631668+0000","last_peered":"2023-10-11T11:19:00.631668+0000","last_clean":"2023-10-11T11:15:27.638303+0000","last_became_active":"2023-10-11T11:15:31.666378+0000","last_became_peered":"2023-10-11T11:15:31.666378+0000","last_unstale":"2023-10-11T11:19:00.631668+0000","last_undegraded":"2023-10-11T11:19:00.631668+0000","last_fullsized":"2023-10-11T11:15:31.649186+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T15:35:19.331209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":516,"reported_epoch":499,"state":"active+undersized","last_fresh":"2023-10-11T11:19:00.636960+0000","last_change":"2023-10-11T11:15:31.679465+0000","last_active":"2023-10-11T11:19:00.636960+0000","last_peered":"2023-10-11T11:19:00.636960+0000","last_clean":"2023-10-11T11:15:27.638749+0000","last_became_active":"2023-10-11T11:15:31.679465+0000","last_became_peered":"2023-10-11T11:15:31.679465+0000","last_unstale":"2023-10-11T11:19:00.636960+0000","last_undegraded":"2023-10-11T11:19:00.636960+0000","last_fullsized":"2023-10-11T11:15:31.659135+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T16:03:42.656692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":11,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738809,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":785348,"kb_used_data":5812,"kb_used_omap":96,"kb_used_meta":779423,"kb_avail":93586492,"statfs":{"total":96636764160,"available":95832567808,"internally_reserved":0,"allocated":5951488,"data_stored":3106918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":98609,"internal_metadata":798129871},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 11:14:37 2023","interfaces":[{"interface":"back","average":{"1min":0.314,"5min":0.316,"15min":0.443},"min":{"1min":0.24399999999999999,"5min":0.14799999999999999,"15min":0.14599999999999999},"max":{"1min":0.48499999999999999,"5min":1.712,"15min":9.6400000000000006},"last":0.31},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.35499999999999998,"15min":0.41299999999999998},"min":{"1min":0.20899999999999999,"5min":0.17199999999999999,"15min":0.16400000000000001},"max":{"1min":3.8180000000000001,"5min":3.8180000000000001,"15min":6.3200000000000003},"last":0.26500000000000001}]},{"osd":1,"last update":"Wed Oct 11 11:19:40 2023","interfaces":[{"interface":"back","average":{"1min":0.29599999999999999,"5min":0.28499999999999998,"15min":0.32300000000000001},"min":{"1min":0.17699999999999999,"5min":0.154,"15min":0.154},"max":{"1min":0.40600000000000003,"5min":0.41999999999999998,"15min":2.9540000000000002},"last":0.312},{"interface":"front","average":{"1min":0.317,"5min":0.31,"15min":0.36199999999999999},"min":{"1min":0.20899999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.45000000000000001,"5min":0.49099999999999999,"15min":3.2719999999999998},"last":0.35099999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738756,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1484496,"kb_used_data":406792,"kb_used_omap":107,"kb_used_meta":1077588,"kb_avail":92887344,"statfs":{"total":96636764160,"available":95116640256,"internally_reserved":0,"allocated":416555008,"data_stored":413648921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110309,"internal_metadata":1103450395},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Oct 11 11:14:30 2023","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.35399999999999998,"15min":0.40600000000000003},"min":{"1min":0.18099999999999999,"5min":0.155,"15min":0.14899999999999999},"max":{"1min":1.4319999999999999,"5min":2.3460000000000001,"15min":6.593},"last":0.26200000000000001},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.39100000000000001,"15min":0.48199999999999998},"min":{"1min":0.189,"5min":0.189,"15min":0.183},"max":{"1min":2.2120000000000002,"5min":2.4089999999999998,"15min":6.5960000000000001},"last":0.316}]},{"osd":2,"last update":"Wed Oct 11 11:14:30 2023","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.35499999999999998,"15min":0.47099999999999997},"min":{"1min":0.20300000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":1.2490000000000001,"5min":2.665,"15min":8.4730000000000008},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.375,"15min":0.442},"min":{"1min":0.23200000000000001,"5min":0.20499999999999999,"15min":0.19600000000000001},"max":{"1min":1.28,"5min":2.6230000000000002,"15min":7.0129999999999999},"last":0.33600000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738810,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":826440,"kb_used_data":6392,"kb_used_omap":88,"kb_used_meta":819943,"kb_avail":93545400,"statfs":{"total":96636764160,"available":95790489600,"internally_reserved":0,"allocated":6545408,"data_stored":3697286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":90523,"internal_metadata":839622245},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 11:14:33 2023","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.42799999999999999,"15min":0.41099999999999998},"min":{"1min":0.151,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":1.2809999999999999,"5min":3.5739999999999998,"15min":8.2479999999999993},"last":0.223},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.48299999999999998,"15min":0.49199999999999999},"min":{"1min":0.20699999999999999,"5min":0.182,"15min":0.17999999999999999},"max":{"1min":1.323,"5min":3.6080000000000001,"15min":8.2870000000000008},"last":0.28399999999999997}]},{"osd":2,"last update":"Wed Oct 11 11:19:41 2023","interfaces":[{"interface":"back","average":{"1min":0.314,"5min":0.35399999999999998,"15min":0.35899999999999999},"min":{"1min":0.186,"5min":0.14699999999999999,"15min":0.14699999999999999},"max":{"1min":0.45500000000000002,"5min":7.4829999999999997,"15min":10.617000000000001},"last":0.33500000000000002},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.39400000000000002,"15min":0.35099999999999998},"min":{"1min":0.21199999999999999,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.49099999999999999,"5min":7.5350000000000001,"15min":7.5350000000000001},"last":0.36899999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-11T11:20:07.753 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~276s 2023-10-11T11:20:10.565 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-10-11T11:20:10.897 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T11:20:10.898 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2023-10-11T11:20:10.910 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":1596,"stamp":"2023-10-11T11:20:10.660522+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":11,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":107,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3096284,"kb_used_data":418996,"kb_used_omap":292,"kb_used_meta":2676955,"kb_avail":280019236,"statfs":{"total":289910292480,"available":286739697664,"internally_reserved":0,"allocated":429051904,"data_stored":420453125,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299441,"internal_metadata":2741202511},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003114"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":517,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-10-11T11:19:02.772828+0000","last_change":"2023-10-11T11:15:31.661747+0000","last_active":"2023-10-11T11:19:02.772828+0000","last_peered":"2023-10-11T11:19:02.772828+0000","last_clean":"2023-10-11T11:15:27.638215+0000","last_became_active":"2023-10-11T11:15:31.661747+0000","last_became_peered":"2023-10-11T11:15:31.661747+0000","last_unstale":"2023-10-11T11:19:02.772828+0000","last_undegraded":"2023-10-11T11:19:02.772828+0000","last_fullsized":"2023-10-11T11:15:31.649336+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:15:25.047039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":29,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-10-11T11:19:02.774213+0000","last_change":"2023-10-11T11:15:31.671363+0000","last_active":"2023-10-11T11:19:02.774213+0000","last_peered":"2023-10-11T11:19:02.774213+0000","last_clean":"2023-10-11T10:43:40.751300+0000","last_became_active":"2023-10-11T11:15:31.671363+0000","last_became_peered":"2023-10-11T11:15:31.671363+0000","last_unstale":"2023-10-11T11:19:02.774213+0000","last_undegraded":"2023-10-11T11:19:02.774213+0000","last_fullsized":"2023-10-11T11:15:31.652001+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:01:53.884964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":512,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-10-11T11:19:03.429640+0000","last_change":"2023-10-11T10:43:41.772781+0000","last_active":"2023-10-11T11:19:03.429640+0000","last_peered":"2023-10-11T11:19:03.429640+0000","last_clean":"2023-10-11T11:19:03.429640+0000","last_became_active":"2023-10-11T10:43:41.772443+0000","last_became_peered":"2023-10-11T10:43:41.772443+0000","last_unstale":"2023-10-11T11:19:03.429640+0000","last_undegraded":"2023-10-11T11:19:03.429640+0000","last_fullsized":"2023-10-11T11:19:03.429640+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T14:15:30.547284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":513,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-10-11T11:19:02.772211+0000","last_change":"2023-10-11T10:43:41.772615+0000","last_active":"2023-10-11T11:19:02.772211+0000","last_peered":"2023-10-11T11:19:02.772211+0000","last_clean":"2023-10-11T11:19:02.772211+0000","last_became_active":"2023-10-11T10:43:41.772290+0000","last_became_peered":"2023-10-11T10:43:41.772290+0000","last_unstale":"2023-10-11T11:19:02.772211+0000","last_undegraded":"2023-10-11T11:19:02.772211+0000","last_fullsized":"2023-10-11T11:19:02.772211+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:26:06.909963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":648,"reported_epoch":500,"state":"active+undersized+degraded","last_fresh":"2023-10-11T11:19:02.769176+0000","last_change":"2023-10-11T11:15:31.654022+0000","last_active":"2023-10-11T11:19:02.769176+0000","last_peered":"2023-10-11T11:19:02.769176+0000","last_clean":"2023-10-11T11:15:27.636884+0000","last_became_active":"2023-10-11T11:15:31.654022+0000","last_became_peered":"2023-10-11T11:15:31.654022+0000","last_unstale":"2023-10-11T11:19:02.769176+0000","last_undegraded":"2023-10-11T11:15:31.646946+0000","last_fullsized":"2023-10-11T11:15:31.646751+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:36.739637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:36.739637+0000","last_clean_scrub_stamp":"2023-10-11T10:43:36.739637+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T11:33:33.696447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":520,"reported_epoch":500,"state":"active+clean+remapped","last_fresh":"2023-10-11T11:19:03.429429+0000","last_change":"2023-10-11T11:15:33.660200+0000","last_active":"2023-10-11T11:19:03.429429+0000","last_peered":"2023-10-11T11:19:03.429429+0000","last_clean":"2023-10-11T11:19:03.429429+0000","last_became_active":"2023-10-11T11:15:33.659619+0000","last_became_peered":"2023-10-11T11:15:33.659619+0000","last_unstale":"2023-10-11T11:19:03.429429+0000","last_undegraded":"2023-10-11T11:19:03.429429+0000","last_fullsized":"2023-10-11T11:19:03.429429+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T17:21:16.866392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":517,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-10-11T11:19:02.775128+0000","last_change":"2023-10-11T11:15:31.670813+0000","last_active":"2023-10-11T11:19:02.775128+0000","last_peered":"2023-10-11T11:19:02.775128+0000","last_clean":"2023-10-11T11:15:27.638219+0000","last_became_active":"2023-10-11T11:15:31.670813+0000","last_became_peered":"2023-10-11T11:15:31.670813+0000","last_unstale":"2023-10-11T11:19:02.775128+0000","last_undegraded":"2023-10-11T11:19:02.775128+0000","last_fullsized":"2023-10-11T11:15:31.658785+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:41:50.872346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":516,"reported_epoch":499,"state":"active+undersized","last_fresh":"2023-10-11T11:19:00.631668+0000","last_change":"2023-10-11T11:15:31.666378+0000","last_active":"2023-10-11T11:19:00.631668+0000","last_peered":"2023-10-11T11:19:00.631668+0000","last_clean":"2023-10-11T11:15:27.638303+0000","last_became_active":"2023-10-11T11:15:31.666378+0000","last_became_peered":"2023-10-11T11:15:31.666378+0000","last_unstale":"2023-10-11T11:19:00.631668+0000","last_undegraded":"2023-10-11T11:19:00.631668+0000","last_fullsized":"2023-10-11T11:15:31.649186+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T15:35:19.331209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":516,"reported_epoch":499,"state":"active+undersized","last_fresh":"2023-10-11T11:19:00.636960+0000","last_change":"2023-10-11T11:15:31.679465+0000","last_active":"2023-10-11T11:19:00.636960+0000","last_peered":"2023-10-11T11:19:00.636960+0000","last_clean":"2023-10-11T11:15:27.638749+0000","last_became_active":"2023-10-11T11:15:31.679465+0000","last_became_peered":"2023-10-11T11:15:31.679465+0000","last_unstale":"2023-10-11T11:19:00.636960+0000","last_undegraded":"2023-10-11T11:19:00.636960+0000","last_fullsized":"2023-10-11T11:15:31.659135+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T16:03:42.656692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":11,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738810,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":785348,"kb_used_data":5812,"kb_used_omap":96,"kb_used_meta":779423,"kb_avail":93586492,"statfs":{"total":96636764160,"available":95832567808,"internally_reserved":0,"allocated":5951488,"data_stored":3106918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":98609,"internal_metadata":798129871},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 11:14:37 2023","interfaces":[{"interface":"back","average":{"1min":0.314,"5min":0.316,"15min":0.443},"min":{"1min":0.24399999999999999,"5min":0.14799999999999999,"15min":0.14599999999999999},"max":{"1min":0.48499999999999999,"5min":1.712,"15min":9.6400000000000006},"last":0.31},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.35499999999999998,"15min":0.41299999999999998},"min":{"1min":0.20899999999999999,"5min":0.17199999999999999,"15min":0.16400000000000001},"max":{"1min":3.8180000000000001,"5min":3.8180000000000001,"15min":6.3200000000000003},"last":0.26500000000000001}]},{"osd":1,"last update":"Wed Oct 11 11:19:40 2023","interfaces":[{"interface":"back","average":{"1min":0.29599999999999999,"5min":0.28499999999999998,"15min":0.32300000000000001},"min":{"1min":0.17699999999999999,"5min":0.154,"15min":0.154},"max":{"1min":0.40600000000000003,"5min":0.41999999999999998,"15min":2.9540000000000002},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.317,"5min":0.31,"15min":0.36199999999999999},"min":{"1min":0.20899999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.45000000000000001,"5min":0.49099999999999999,"15min":3.2719999999999998},"last":0.51900000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738756,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1484496,"kb_used_data":406792,"kb_used_omap":107,"kb_used_meta":1077588,"kb_avail":92887344,"statfs":{"total":96636764160,"available":95116640256,"internally_reserved":0,"allocated":416555008,"data_stored":413648921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110309,"internal_metadata":1103450395},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Oct 11 11:14:30 2023","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.35399999999999998,"15min":0.40600000000000003},"min":{"1min":0.18099999999999999,"5min":0.155,"15min":0.14899999999999999},"max":{"1min":1.4319999999999999,"5min":2.3460000000000001,"15min":6.593},"last":0.26200000000000001},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.39100000000000001,"15min":0.48199999999999998},"min":{"1min":0.189,"5min":0.189,"15min":0.183},"max":{"1min":2.2120000000000002,"5min":2.4089999999999998,"15min":6.5960000000000001},"last":0.316}]},{"osd":2,"last update":"Wed Oct 11 11:14:30 2023","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.35499999999999998,"15min":0.47099999999999997},"min":{"1min":0.20300000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":1.2490000000000001,"5min":2.665,"15min":8.4730000000000008},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.375,"15min":0.442},"min":{"1min":0.23200000000000001,"5min":0.20499999999999999,"15min":0.19600000000000001},"max":{"1min":1.28,"5min":2.6230000000000002,"15min":7.0129999999999999},"last":0.33600000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738811,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":826440,"kb_used_data":6392,"kb_used_omap":88,"kb_used_meta":819943,"kb_avail":93545400,"statfs":{"total":96636764160,"available":95790489600,"internally_reserved":0,"allocated":6545408,"data_stored":3697286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":90523,"internal_metadata":839622245},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 11:14:33 2023","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.42799999999999999,"15min":0.41099999999999998},"min":{"1min":0.151,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":1.2809999999999999,"5min":3.5739999999999998,"15min":8.2479999999999993},"last":0.223},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.48299999999999998,"15min":0.49199999999999999},"min":{"1min":0.20699999999999999,"5min":0.182,"15min":0.17999999999999999},"max":{"1min":1.323,"5min":3.6080000000000001,"15min":8.2870000000000008},"last":0.28399999999999997}]},{"osd":2,"last update":"Wed Oct 11 11:19:41 2023","interfaces":[{"interface":"back","average":{"1min":0.314,"5min":0.35399999999999998,"15min":0.35899999999999999},"min":{"1min":0.186,"5min":0.14699999999999999,"15min":0.14699999999999999},"max":{"1min":0.45500000000000002,"5min":7.4829999999999997,"15min":10.617000000000001},"last":0.30399999999999999},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.39400000000000002,"15min":0.35099999999999998},"min":{"1min":0.21199999999999999,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.49099999999999999,"5min":7.5350000000000001,"15min":7.5350000000000001},"last":0.34200000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-11T11:20:10.911 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-10-11T11:20:11.226 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T11:20:11.228 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2023-10-11T11:20:11.240 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":1596,"stamp":"2023-10-11T11:20:10.660522+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":11,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":107,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3096284,"kb_used_data":418996,"kb_used_omap":292,"kb_used_meta":2676955,"kb_avail":280019236,"statfs":{"total":289910292480,"available":286739697664,"internally_reserved":0,"allocated":429051904,"data_stored":420453125,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299441,"internal_metadata":2741202511},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003114"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":517,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-10-11T11:19:02.772828+0000","last_change":"2023-10-11T11:15:31.661747+0000","last_active":"2023-10-11T11:19:02.772828+0000","last_peered":"2023-10-11T11:19:02.772828+0000","last_clean":"2023-10-11T11:15:27.638215+0000","last_became_active":"2023-10-11T11:15:31.661747+0000","last_became_peered":"2023-10-11T11:15:31.661747+0000","last_unstale":"2023-10-11T11:19:02.772828+0000","last_undegraded":"2023-10-11T11:19:02.772828+0000","last_fullsized":"2023-10-11T11:15:31.649336+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:15:25.047039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":29,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-10-11T11:19:02.774213+0000","last_change":"2023-10-11T11:15:31.671363+0000","last_active":"2023-10-11T11:19:02.774213+0000","last_peered":"2023-10-11T11:19:02.774213+0000","last_clean":"2023-10-11T10:43:40.751300+0000","last_became_active":"2023-10-11T11:15:31.671363+0000","last_became_peered":"2023-10-11T11:15:31.671363+0000","last_unstale":"2023-10-11T11:19:02.774213+0000","last_undegraded":"2023-10-11T11:19:02.774213+0000","last_fullsized":"2023-10-11T11:15:31.652001+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:01:53.884964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":512,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-10-11T11:19:03.429640+0000","last_change":"2023-10-11T10:43:41.772781+0000","last_active":"2023-10-11T11:19:03.429640+0000","last_peered":"2023-10-11T11:19:03.429640+0000","last_clean":"2023-10-11T11:19:03.429640+0000","last_became_active":"2023-10-11T10:43:41.772443+0000","last_became_peered":"2023-10-11T10:43:41.772443+0000","last_unstale":"2023-10-11T11:19:03.429640+0000","last_undegraded":"2023-10-11T11:19:03.429640+0000","last_fullsized":"2023-10-11T11:19:03.429640+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T14:15:30.547284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":513,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-10-11T11:19:02.772211+0000","last_change":"2023-10-11T10:43:41.772615+0000","last_active":"2023-10-11T11:19:02.772211+0000","last_peered":"2023-10-11T11:19:02.772211+0000","last_clean":"2023-10-11T11:19:02.772211+0000","last_became_active":"2023-10-11T10:43:41.772290+0000","last_became_peered":"2023-10-11T10:43:41.772290+0000","last_unstale":"2023-10-11T11:19:02.772211+0000","last_undegraded":"2023-10-11T11:19:02.772211+0000","last_fullsized":"2023-10-11T11:19:02.772211+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:26:06.909963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":648,"reported_epoch":500,"state":"active+undersized+degraded","last_fresh":"2023-10-11T11:19:02.769176+0000","last_change":"2023-10-11T11:15:31.654022+0000","last_active":"2023-10-11T11:19:02.769176+0000","last_peered":"2023-10-11T11:19:02.769176+0000","last_clean":"2023-10-11T11:15:27.636884+0000","last_became_active":"2023-10-11T11:15:31.654022+0000","last_became_peered":"2023-10-11T11:15:31.654022+0000","last_unstale":"2023-10-11T11:19:02.769176+0000","last_undegraded":"2023-10-11T11:15:31.646946+0000","last_fullsized":"2023-10-11T11:15:31.646751+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:36.739637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:36.739637+0000","last_clean_scrub_stamp":"2023-10-11T10:43:36.739637+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T11:33:33.696447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":520,"reported_epoch":500,"state":"active+clean+remapped","last_fresh":"2023-10-11T11:19:03.429429+0000","last_change":"2023-10-11T11:15:33.660200+0000","last_active":"2023-10-11T11:19:03.429429+0000","last_peered":"2023-10-11T11:19:03.429429+0000","last_clean":"2023-10-11T11:19:03.429429+0000","last_became_active":"2023-10-11T11:15:33.659619+0000","last_became_peered":"2023-10-11T11:15:33.659619+0000","last_unstale":"2023-10-11T11:19:03.429429+0000","last_undegraded":"2023-10-11T11:19:03.429429+0000","last_fullsized":"2023-10-11T11:19:03.429429+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T17:21:16.866392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":517,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-10-11T11:19:02.775128+0000","last_change":"2023-10-11T11:15:31.670813+0000","last_active":"2023-10-11T11:19:02.775128+0000","last_peered":"2023-10-11T11:19:02.775128+0000","last_clean":"2023-10-11T11:15:27.638219+0000","last_became_active":"2023-10-11T11:15:31.670813+0000","last_became_peered":"2023-10-11T11:15:31.670813+0000","last_unstale":"2023-10-11T11:19:02.775128+0000","last_undegraded":"2023-10-11T11:19:02.775128+0000","last_fullsized":"2023-10-11T11:15:31.658785+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:41:50.872346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":516,"reported_epoch":499,"state":"active+undersized","last_fresh":"2023-10-11T11:19:00.631668+0000","last_change":"2023-10-11T11:15:31.666378+0000","last_active":"2023-10-11T11:19:00.631668+0000","last_peered":"2023-10-11T11:19:00.631668+0000","last_clean":"2023-10-11T11:15:27.638303+0000","last_became_active":"2023-10-11T11:15:31.666378+0000","last_became_peered":"2023-10-11T11:15:31.666378+0000","last_unstale":"2023-10-11T11:19:00.631668+0000","last_undegraded":"2023-10-11T11:19:00.631668+0000","last_fullsized":"2023-10-11T11:15:31.649186+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T15:35:19.331209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":516,"reported_epoch":499,"state":"active+undersized","last_fresh":"2023-10-11T11:19:00.636960+0000","last_change":"2023-10-11T11:15:31.679465+0000","last_active":"2023-10-11T11:19:00.636960+0000","last_peered":"2023-10-11T11:19:00.636960+0000","last_clean":"2023-10-11T11:15:27.638749+0000","last_became_active":"2023-10-11T11:15:31.679465+0000","last_became_peered":"2023-10-11T11:15:31.679465+0000","last_unstale":"2023-10-11T11:19:00.636960+0000","last_undegraded":"2023-10-11T11:19:00.636960+0000","last_fullsized":"2023-10-11T11:15:31.659135+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T16:03:42.656692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":11,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738810,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":785348,"kb_used_data":5812,"kb_used_omap":96,"kb_used_meta":779423,"kb_avail":93586492,"statfs":{"total":96636764160,"available":95832567808,"internally_reserved":0,"allocated":5951488,"data_stored":3106918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":98609,"internal_metadata":798129871},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 11:14:37 2023","interfaces":[{"interface":"back","average":{"1min":0.314,"5min":0.316,"15min":0.443},"min":{"1min":0.24399999999999999,"5min":0.14799999999999999,"15min":0.14599999999999999},"max":{"1min":0.48499999999999999,"5min":1.712,"15min":9.6400000000000006},"last":0.31},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.35499999999999998,"15min":0.41299999999999998},"min":{"1min":0.20899999999999999,"5min":0.17199999999999999,"15min":0.16400000000000001},"max":{"1min":3.8180000000000001,"5min":3.8180000000000001,"15min":6.3200000000000003},"last":0.26500000000000001}]},{"osd":1,"last update":"Wed Oct 11 11:19:40 2023","interfaces":[{"interface":"back","average":{"1min":0.29599999999999999,"5min":0.28499999999999998,"15min":0.32300000000000001},"min":{"1min":0.17699999999999999,"5min":0.154,"15min":0.154},"max":{"1min":0.40600000000000003,"5min":0.41999999999999998,"15min":2.9540000000000002},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.317,"5min":0.31,"15min":0.36199999999999999},"min":{"1min":0.20899999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.45000000000000001,"5min":0.49099999999999999,"15min":3.2719999999999998},"last":0.51900000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738756,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1484496,"kb_used_data":406792,"kb_used_omap":107,"kb_used_meta":1077588,"kb_avail":92887344,"statfs":{"total":96636764160,"available":95116640256,"internally_reserved":0,"allocated":416555008,"data_stored":413648921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110309,"internal_metadata":1103450395},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Oct 11 11:14:30 2023","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.35399999999999998,"15min":0.40600000000000003},"min":{"1min":0.18099999999999999,"5min":0.155,"15min":0.14899999999999999},"max":{"1min":1.4319999999999999,"5min":2.3460000000000001,"15min":6.593},"last":0.26200000000000001},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.39100000000000001,"15min":0.48199999999999998},"min":{"1min":0.189,"5min":0.189,"15min":0.183},"max":{"1min":2.2120000000000002,"5min":2.4089999999999998,"15min":6.5960000000000001},"last":0.316}]},{"osd":2,"last update":"Wed Oct 11 11:14:30 2023","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.35499999999999998,"15min":0.47099999999999997},"min":{"1min":0.20300000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":1.2490000000000001,"5min":2.665,"15min":8.4730000000000008},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.375,"15min":0.442},"min":{"1min":0.23200000000000001,"5min":0.20499999999999999,"15min":0.19600000000000001},"max":{"1min":1.28,"5min":2.6230000000000002,"15min":7.0129999999999999},"last":0.33600000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738811,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":826440,"kb_used_data":6392,"kb_used_omap":88,"kb_used_meta":819943,"kb_avail":93545400,"statfs":{"total":96636764160,"available":95790489600,"internally_reserved":0,"allocated":6545408,"data_stored":3697286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":90523,"internal_metadata":839622245},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 11:14:33 2023","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.42799999999999999,"15min":0.41099999999999998},"min":{"1min":0.151,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":1.2809999999999999,"5min":3.5739999999999998,"15min":8.2479999999999993},"last":0.223},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.48299999999999998,"15min":0.49199999999999999},"min":{"1min":0.20699999999999999,"5min":0.182,"15min":0.17999999999999999},"max":{"1min":1.323,"5min":3.6080000000000001,"15min":8.2870000000000008},"last":0.28399999999999997}]},{"osd":2,"last update":"Wed Oct 11 11:19:41 2023","interfaces":[{"interface":"back","average":{"1min":0.314,"5min":0.35399999999999998,"15min":0.35899999999999999},"min":{"1min":0.186,"5min":0.14699999999999999,"15min":0.14699999999999999},"max":{"1min":0.45500000000000002,"5min":7.4829999999999997,"15min":10.617000000000001},"last":0.30399999999999999},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.39400000000000002,"15min":0.35099999999999998},"min":{"1min":0.21199999999999999,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.49099999999999999,"5min":7.5350000000000001,"15min":7.5350000000000001},"last":0.34200000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-11T11:20:11.241 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-10-11T11:20:11.241 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 517, 'reported_epoch': 500, 'state': 'active+undersized', 'last_fresh': '2023-10-11T11:19:02.772828+0000', 'last_change': '2023-10-11T11:15:31.661747+0000', 'last_active': '2023-10-11T11:19:02.772828+0000', 'last_peered': '2023-10-11T11:19:02.772828+0000', 'last_clean': '2023-10-11T11:15:27.638215+0000', 'last_became_active': '2023-10-11T11:15:31.661747+0000', 'last_became_peered': '2023-10-11T11:15:31.661747+0000', 'last_unstale': '2023-10-11T11:19:02.772828+0000', 'last_undegraded': '2023-10-11T11:19:02.772828+0000', 'last_fullsized': '2023-10-11T11:15:31.649336+0000', 'mapping_epoch': 480, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_clean_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-12T22:15:25.047039+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-10-11T11:20:11.242 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-10-11T11:20:11.242 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 29, 'reported_epoch': 500, 'state': 'active+undersized', 'last_fresh': '2023-10-11T11:19:02.774213+0000', 'last_change': '2023-10-11T11:15:31.671363+0000', 'last_active': '2023-10-11T11:19:02.774213+0000', 'last_peered': '2023-10-11T11:19:02.774213+0000', 'last_clean': '2023-10-11T10:43:40.751300+0000', 'last_became_active': '2023-10-11T11:15:31.671363+0000', 'last_became_peered': '2023-10-11T11:15:31.671363+0000', 'last_unstale': '2023-10-11T11:19:02.774213+0000', 'last_undegraded': '2023-10-11T11:19:02.774213+0000', 'last_fullsized': '2023-10-11T11:15:31.652001+0000', 'mapping_epoch': 480, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_clean_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-12T21:01:53.884964+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-10-11T11:20:11.243 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-10-11T11:20:11.243 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "11'97", 'reported_seq': 648, 'reported_epoch': 500, 'state': 'active+undersized+degraded', 'last_fresh': '2023-10-11T11:19:02.769176+0000', 'last_change': '2023-10-11T11:15:31.654022+0000', 'last_active': '2023-10-11T11:19:02.769176+0000', 'last_peered': '2023-10-11T11:19:02.769176+0000', 'last_clean': '2023-10-11T11:15:27.636884+0000', 'last_became_active': '2023-10-11T11:15:31.654022+0000', 'last_became_peered': '2023-10-11T11:15:31.654022+0000', 'last_unstale': '2023-10-11T11:19:02.769176+0000', 'last_undegraded': '2023-10-11T11:15:31.646946+0000', 'last_fullsized': '2023-10-11T11:15:31.646751+0000', 'mapping_epoch': 480, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-11T10:43:36.739637+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-11T10:43:36.739637+0000', 'last_clean_scrub_stamp': '2023-10-11T10:43:36.739637+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-12T11:33:33.696447+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-10-11T11:20:11.243 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.0 is not active+clean 2023-10-11T11:20:11.243 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 520, 'reported_epoch': 500, 'state': 'active+clean+remapped', 'last_fresh': '2023-10-11T11:19:03.429429+0000', 'last_change': '2023-10-11T11:15:33.660200+0000', 'last_active': '2023-10-11T11:19:03.429429+0000', 'last_peered': '2023-10-11T11:19:03.429429+0000', 'last_clean': '2023-10-11T11:19:03.429429+0000', 'last_became_active': '2023-10-11T11:15:33.659619+0000', 'last_became_peered': '2023-10-11T11:15:33.659619+0000', 'last_unstale': '2023-10-11T11:19:03.429429+0000', 'last_undegraded': '2023-10-11T11:19:03.429429+0000', 'last_fullsized': '2023-10-11T11:19:03.429429+0000', 'mapping_epoch': 482, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 483, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_clean_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-12T17:21:16.866392+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2, 1], 'avail_no_missing': ['2', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-10-11T11:20:11.244 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-10-11T11:20:11.244 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 517, 'reported_epoch': 500, 'state': 'active+undersized', 'last_fresh': '2023-10-11T11:19:02.775128+0000', 'last_change': '2023-10-11T11:15:31.670813+0000', 'last_active': '2023-10-11T11:19:02.775128+0000', 'last_peered': '2023-10-11T11:19:02.775128+0000', 'last_clean': '2023-10-11T11:15:27.638219+0000', 'last_became_active': '2023-10-11T11:15:31.670813+0000', 'last_became_peered': '2023-10-11T11:15:31.670813+0000', 'last_unstale': '2023-10-11T11:19:02.775128+0000', 'last_undegraded': '2023-10-11T11:19:02.775128+0000', 'last_fullsized': '2023-10-11T11:15:31.658785+0000', 'mapping_epoch': 480, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_clean_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-12T21:41:50.872346+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-10-11T11:20:11.244 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-10-11T11:20:11.245 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 516, 'reported_epoch': 499, 'state': 'active+undersized', 'last_fresh': '2023-10-11T11:19:00.631668+0000', 'last_change': '2023-10-11T11:15:31.666378+0000', 'last_active': '2023-10-11T11:19:00.631668+0000', 'last_peered': '2023-10-11T11:19:00.631668+0000', 'last_clean': '2023-10-11T11:15:27.638303+0000', 'last_became_active': '2023-10-11T11:15:31.666378+0000', 'last_became_peered': '2023-10-11T11:15:31.666378+0000', 'last_unstale': '2023-10-11T11:19:00.631668+0000', 'last_undegraded': '2023-10-11T11:19:00.631668+0000', 'last_fullsized': '2023-10-11T11:15:31.649186+0000', 'mapping_epoch': 480, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_clean_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-12T15:35:19.331209+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-10-11T11:20:11.245 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-10-11T11:20:11.245 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 516, 'reported_epoch': 499, 'state': 'active+undersized', 'last_fresh': '2023-10-11T11:19:00.636960+0000', 'last_change': '2023-10-11T11:15:31.679465+0000', 'last_active': '2023-10-11T11:19:00.636960+0000', 'last_peered': '2023-10-11T11:19:00.636960+0000', 'last_clean': '2023-10-11T11:15:27.638749+0000', 'last_became_active': '2023-10-11T11:15:31.679465+0000', 'last_became_peered': '2023-10-11T11:15:31.679465+0000', 'last_unstale': '2023-10-11T11:19:00.636960+0000', 'last_undegraded': '2023-10-11T11:19:00.636960+0000', 'last_fullsized': '2023-10-11T11:15:31.659135+0000', 'mapping_epoch': 480, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_clean_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-12T16:03:42.656692+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-10-11T11:20:11.246 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-10-11T11:20:11.643 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T11:20:11.643 INFO:teuthology.orchestra.run.smithi111.stdout:{"fsid":"022ce6f2-9a5b-45ad-8b77-a1332d92f228","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2206,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":500,"num_osds":3,"num_up_osds":2,"osd_up_since":1697022930,"num_in_osds":3,"osd_in_since":1697021005,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":2},{"state_name":"active+undersized+degraded","count":1},{"state_name":"active+clean+remapped","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3170594816,"bytes_avail":286739697664,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":36,"modified":"2023-10-11T11:19:04.644408+0000","services":{}},"progress_events":{}} 2023-10-11T11:20:11.657 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-10-11T11:20:11.657 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-10-11T11:20:11.977 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T11:20:11.979 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2023-10-11T11:20:11.990 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":1596,"stamp":"2023-10-11T11:20:10.660522+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":11,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":107,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3096284,"kb_used_data":418996,"kb_used_omap":292,"kb_used_meta":2676955,"kb_avail":280019236,"statfs":{"total":289910292480,"available":286739697664,"internally_reserved":0,"allocated":429051904,"data_stored":420453125,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299441,"internal_metadata":2741202511},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003114"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":517,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-10-11T11:19:02.772828+0000","last_change":"2023-10-11T11:15:31.661747+0000","last_active":"2023-10-11T11:19:02.772828+0000","last_peered":"2023-10-11T11:19:02.772828+0000","last_clean":"2023-10-11T11:15:27.638215+0000","last_became_active":"2023-10-11T11:15:31.661747+0000","last_became_peered":"2023-10-11T11:15:31.661747+0000","last_unstale":"2023-10-11T11:19:02.772828+0000","last_undegraded":"2023-10-11T11:19:02.772828+0000","last_fullsized":"2023-10-11T11:15:31.649336+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:15:25.047039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":29,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-10-11T11:19:02.774213+0000","last_change":"2023-10-11T11:15:31.671363+0000","last_active":"2023-10-11T11:19:02.774213+0000","last_peered":"2023-10-11T11:19:02.774213+0000","last_clean":"2023-10-11T10:43:40.751300+0000","last_became_active":"2023-10-11T11:15:31.671363+0000","last_became_peered":"2023-10-11T11:15:31.671363+0000","last_unstale":"2023-10-11T11:19:02.774213+0000","last_undegraded":"2023-10-11T11:19:02.774213+0000","last_fullsized":"2023-10-11T11:15:31.652001+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:01:53.884964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":512,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-10-11T11:19:03.429640+0000","last_change":"2023-10-11T10:43:41.772781+0000","last_active":"2023-10-11T11:19:03.429640+0000","last_peered":"2023-10-11T11:19:03.429640+0000","last_clean":"2023-10-11T11:19:03.429640+0000","last_became_active":"2023-10-11T10:43:41.772443+0000","last_became_peered":"2023-10-11T10:43:41.772443+0000","last_unstale":"2023-10-11T11:19:03.429640+0000","last_undegraded":"2023-10-11T11:19:03.429640+0000","last_fullsized":"2023-10-11T11:19:03.429640+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T14:15:30.547284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":513,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-10-11T11:19:02.772211+0000","last_change":"2023-10-11T10:43:41.772615+0000","last_active":"2023-10-11T11:19:02.772211+0000","last_peered":"2023-10-11T11:19:02.772211+0000","last_clean":"2023-10-11T11:19:02.772211+0000","last_became_active":"2023-10-11T10:43:41.772290+0000","last_became_peered":"2023-10-11T10:43:41.772290+0000","last_unstale":"2023-10-11T11:19:02.772211+0000","last_undegraded":"2023-10-11T11:19:02.772211+0000","last_fullsized":"2023-10-11T11:19:02.772211+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:26:06.909963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":648,"reported_epoch":500,"state":"active+undersized+degraded","last_fresh":"2023-10-11T11:19:02.769176+0000","last_change":"2023-10-11T11:15:31.654022+0000","last_active":"2023-10-11T11:19:02.769176+0000","last_peered":"2023-10-11T11:19:02.769176+0000","last_clean":"2023-10-11T11:15:27.636884+0000","last_became_active":"2023-10-11T11:15:31.654022+0000","last_became_peered":"2023-10-11T11:15:31.654022+0000","last_unstale":"2023-10-11T11:19:02.769176+0000","last_undegraded":"2023-10-11T11:15:31.646946+0000","last_fullsized":"2023-10-11T11:15:31.646751+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:36.739637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:36.739637+0000","last_clean_scrub_stamp":"2023-10-11T10:43:36.739637+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T11:33:33.696447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":520,"reported_epoch":500,"state":"active+clean+remapped","last_fresh":"2023-10-11T11:19:03.429429+0000","last_change":"2023-10-11T11:15:33.660200+0000","last_active":"2023-10-11T11:19:03.429429+0000","last_peered":"2023-10-11T11:19:03.429429+0000","last_clean":"2023-10-11T11:19:03.429429+0000","last_became_active":"2023-10-11T11:15:33.659619+0000","last_became_peered":"2023-10-11T11:15:33.659619+0000","last_unstale":"2023-10-11T11:19:03.429429+0000","last_undegraded":"2023-10-11T11:19:03.429429+0000","last_fullsized":"2023-10-11T11:19:03.429429+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T17:21:16.866392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":517,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-10-11T11:19:02.775128+0000","last_change":"2023-10-11T11:15:31.670813+0000","last_active":"2023-10-11T11:19:02.775128+0000","last_peered":"2023-10-11T11:19:02.775128+0000","last_clean":"2023-10-11T11:15:27.638219+0000","last_became_active":"2023-10-11T11:15:31.670813+0000","last_became_peered":"2023-10-11T11:15:31.670813+0000","last_unstale":"2023-10-11T11:19:02.775128+0000","last_undegraded":"2023-10-11T11:19:02.775128+0000","last_fullsized":"2023-10-11T11:15:31.658785+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:41:50.872346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":516,"reported_epoch":499,"state":"active+undersized","last_fresh":"2023-10-11T11:19:00.631668+0000","last_change":"2023-10-11T11:15:31.666378+0000","last_active":"2023-10-11T11:19:00.631668+0000","last_peered":"2023-10-11T11:19:00.631668+0000","last_clean":"2023-10-11T11:15:27.638303+0000","last_became_active":"2023-10-11T11:15:31.666378+0000","last_became_peered":"2023-10-11T11:15:31.666378+0000","last_unstale":"2023-10-11T11:19:00.631668+0000","last_undegraded":"2023-10-11T11:19:00.631668+0000","last_fullsized":"2023-10-11T11:15:31.649186+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T15:35:19.331209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":516,"reported_epoch":499,"state":"active+undersized","last_fresh":"2023-10-11T11:19:00.636960+0000","last_change":"2023-10-11T11:15:31.679465+0000","last_active":"2023-10-11T11:19:00.636960+0000","last_peered":"2023-10-11T11:19:00.636960+0000","last_clean":"2023-10-11T11:15:27.638749+0000","last_became_active":"2023-10-11T11:15:31.679465+0000","last_became_peered":"2023-10-11T11:15:31.679465+0000","last_unstale":"2023-10-11T11:19:00.636960+0000","last_undegraded":"2023-10-11T11:19:00.636960+0000","last_fullsized":"2023-10-11T11:15:31.659135+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T16:03:42.656692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":11,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738810,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":785348,"kb_used_data":5812,"kb_used_omap":96,"kb_used_meta":779423,"kb_avail":93586492,"statfs":{"total":96636764160,"available":95832567808,"internally_reserved":0,"allocated":5951488,"data_stored":3106918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":98609,"internal_metadata":798129871},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 11:14:37 2023","interfaces":[{"interface":"back","average":{"1min":0.314,"5min":0.316,"15min":0.443},"min":{"1min":0.24399999999999999,"5min":0.14799999999999999,"15min":0.14599999999999999},"max":{"1min":0.48499999999999999,"5min":1.712,"15min":9.6400000000000006},"last":0.31},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.35499999999999998,"15min":0.41299999999999998},"min":{"1min":0.20899999999999999,"5min":0.17199999999999999,"15min":0.16400000000000001},"max":{"1min":3.8180000000000001,"5min":3.8180000000000001,"15min":6.3200000000000003},"last":0.26500000000000001}]},{"osd":1,"last update":"Wed Oct 11 11:19:40 2023","interfaces":[{"interface":"back","average":{"1min":0.29599999999999999,"5min":0.28499999999999998,"15min":0.32300000000000001},"min":{"1min":0.17699999999999999,"5min":0.154,"15min":0.154},"max":{"1min":0.40600000000000003,"5min":0.41999999999999998,"15min":2.9540000000000002},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.317,"5min":0.31,"15min":0.36199999999999999},"min":{"1min":0.20899999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.45000000000000001,"5min":0.49099999999999999,"15min":3.2719999999999998},"last":0.51900000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738756,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1484496,"kb_used_data":406792,"kb_used_omap":107,"kb_used_meta":1077588,"kb_avail":92887344,"statfs":{"total":96636764160,"available":95116640256,"internally_reserved":0,"allocated":416555008,"data_stored":413648921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110309,"internal_metadata":1103450395},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Oct 11 11:14:30 2023","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.35399999999999998,"15min":0.40600000000000003},"min":{"1min":0.18099999999999999,"5min":0.155,"15min":0.14899999999999999},"max":{"1min":1.4319999999999999,"5min":2.3460000000000001,"15min":6.593},"last":0.26200000000000001},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.39100000000000001,"15min":0.48199999999999998},"min":{"1min":0.189,"5min":0.189,"15min":0.183},"max":{"1min":2.2120000000000002,"5min":2.4089999999999998,"15min":6.5960000000000001},"last":0.316}]},{"osd":2,"last update":"Wed Oct 11 11:14:30 2023","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.35499999999999998,"15min":0.47099999999999997},"min":{"1min":0.20300000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":1.2490000000000001,"5min":2.665,"15min":8.4730000000000008},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.375,"15min":0.442},"min":{"1min":0.23200000000000001,"5min":0.20499999999999999,"15min":0.19600000000000001},"max":{"1min":1.28,"5min":2.6230000000000002,"15min":7.0129999999999999},"last":0.33600000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738811,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":826440,"kb_used_data":6392,"kb_used_omap":88,"kb_used_meta":819943,"kb_avail":93545400,"statfs":{"total":96636764160,"available":95790489600,"internally_reserved":0,"allocated":6545408,"data_stored":3697286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":90523,"internal_metadata":839622245},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 11:14:33 2023","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.42799999999999999,"15min":0.41099999999999998},"min":{"1min":0.151,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":1.2809999999999999,"5min":3.5739999999999998,"15min":8.2479999999999993},"last":0.223},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.48299999999999998,"15min":0.49199999999999999},"min":{"1min":0.20699999999999999,"5min":0.182,"15min":0.17999999999999999},"max":{"1min":1.323,"5min":3.6080000000000001,"15min":8.2870000000000008},"last":0.28399999999999997}]},{"osd":2,"last update":"Wed Oct 11 11:19:41 2023","interfaces":[{"interface":"back","average":{"1min":0.314,"5min":0.35399999999999998,"15min":0.35899999999999999},"min":{"1min":0.186,"5min":0.14699999999999999,"15min":0.14699999999999999},"max":{"1min":0.45500000000000002,"5min":7.4829999999999997,"15min":10.617000000000001},"last":0.30399999999999999},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.39400000000000002,"15min":0.35099999999999998},"min":{"1min":0.21199999999999999,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.49099999999999999,"5min":7.5350000000000001,"15min":7.5350000000000001},"last":0.34200000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-11T11:20:13.158 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~281s 2023-10-11T11:20:14.992 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-10-11T11:20:15.322 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T11:20:15.324 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2023-10-11T11:20:15.336 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":1598,"stamp":"2023-10-11T11:20:14.661430+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":11,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":107,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3096284,"kb_used_data":418996,"kb_used_omap":292,"kb_used_meta":2676955,"kb_avail":280019236,"statfs":{"total":289910292480,"available":286739697664,"internally_reserved":0,"allocated":429051904,"data_stored":420453125,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299441,"internal_metadata":2741202511},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003261"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":517,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-10-11T11:19:02.772828+0000","last_change":"2023-10-11T11:15:31.661747+0000","last_active":"2023-10-11T11:19:02.772828+0000","last_peered":"2023-10-11T11:19:02.772828+0000","last_clean":"2023-10-11T11:15:27.638215+0000","last_became_active":"2023-10-11T11:15:31.661747+0000","last_became_peered":"2023-10-11T11:15:31.661747+0000","last_unstale":"2023-10-11T11:19:02.772828+0000","last_undegraded":"2023-10-11T11:19:02.772828+0000","last_fullsized":"2023-10-11T11:15:31.649336+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:15:25.047039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":29,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-10-11T11:19:02.774213+0000","last_change":"2023-10-11T11:15:31.671363+0000","last_active":"2023-10-11T11:19:02.774213+0000","last_peered":"2023-10-11T11:19:02.774213+0000","last_clean":"2023-10-11T10:43:40.751300+0000","last_became_active":"2023-10-11T11:15:31.671363+0000","last_became_peered":"2023-10-11T11:15:31.671363+0000","last_unstale":"2023-10-11T11:19:02.774213+0000","last_undegraded":"2023-10-11T11:19:02.774213+0000","last_fullsized":"2023-10-11T11:15:31.652001+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:01:53.884964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":512,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-10-11T11:19:03.429640+0000","last_change":"2023-10-11T10:43:41.772781+0000","last_active":"2023-10-11T11:19:03.429640+0000","last_peered":"2023-10-11T11:19:03.429640+0000","last_clean":"2023-10-11T11:19:03.429640+0000","last_became_active":"2023-10-11T10:43:41.772443+0000","last_became_peered":"2023-10-11T10:43:41.772443+0000","last_unstale":"2023-10-11T11:19:03.429640+0000","last_undegraded":"2023-10-11T11:19:03.429640+0000","last_fullsized":"2023-10-11T11:19:03.429640+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T14:15:30.547284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":513,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-10-11T11:19:02.772211+0000","last_change":"2023-10-11T10:43:41.772615+0000","last_active":"2023-10-11T11:19:02.772211+0000","last_peered":"2023-10-11T11:19:02.772211+0000","last_clean":"2023-10-11T11:19:02.772211+0000","last_became_active":"2023-10-11T10:43:41.772290+0000","last_became_peered":"2023-10-11T10:43:41.772290+0000","last_unstale":"2023-10-11T11:19:02.772211+0000","last_undegraded":"2023-10-11T11:19:02.772211+0000","last_fullsized":"2023-10-11T11:19:02.772211+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:26:06.909963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":648,"reported_epoch":500,"state":"active+undersized+degraded","last_fresh":"2023-10-11T11:19:02.769176+0000","last_change":"2023-10-11T11:15:31.654022+0000","last_active":"2023-10-11T11:19:02.769176+0000","last_peered":"2023-10-11T11:19:02.769176+0000","last_clean":"2023-10-11T11:15:27.636884+0000","last_became_active":"2023-10-11T11:15:31.654022+0000","last_became_peered":"2023-10-11T11:15:31.654022+0000","last_unstale":"2023-10-11T11:19:02.769176+0000","last_undegraded":"2023-10-11T11:15:31.646946+0000","last_fullsized":"2023-10-11T11:15:31.646751+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:36.739637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:36.739637+0000","last_clean_scrub_stamp":"2023-10-11T10:43:36.739637+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T11:33:33.696447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":520,"reported_epoch":500,"state":"active+clean+remapped","last_fresh":"2023-10-11T11:19:03.429429+0000","last_change":"2023-10-11T11:15:33.660200+0000","last_active":"2023-10-11T11:19:03.429429+0000","last_peered":"2023-10-11T11:19:03.429429+0000","last_clean":"2023-10-11T11:19:03.429429+0000","last_became_active":"2023-10-11T11:15:33.659619+0000","last_became_peered":"2023-10-11T11:15:33.659619+0000","last_unstale":"2023-10-11T11:19:03.429429+0000","last_undegraded":"2023-10-11T11:19:03.429429+0000","last_fullsized":"2023-10-11T11:19:03.429429+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T17:21:16.866392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":517,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-10-11T11:19:02.775128+0000","last_change":"2023-10-11T11:15:31.670813+0000","last_active":"2023-10-11T11:19:02.775128+0000","last_peered":"2023-10-11T11:19:02.775128+0000","last_clean":"2023-10-11T11:15:27.638219+0000","last_became_active":"2023-10-11T11:15:31.670813+0000","last_became_peered":"2023-10-11T11:15:31.670813+0000","last_unstale":"2023-10-11T11:19:02.775128+0000","last_undegraded":"2023-10-11T11:19:02.775128+0000","last_fullsized":"2023-10-11T11:15:31.658785+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:41:50.872346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":516,"reported_epoch":499,"state":"active+undersized","last_fresh":"2023-10-11T11:19:00.631668+0000","last_change":"2023-10-11T11:15:31.666378+0000","last_active":"2023-10-11T11:19:00.631668+0000","last_peered":"2023-10-11T11:19:00.631668+0000","last_clean":"2023-10-11T11:15:27.638303+0000","last_became_active":"2023-10-11T11:15:31.666378+0000","last_became_peered":"2023-10-11T11:15:31.666378+0000","last_unstale":"2023-10-11T11:19:00.631668+0000","last_undegraded":"2023-10-11T11:19:00.631668+0000","last_fullsized":"2023-10-11T11:15:31.649186+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T15:35:19.331209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":516,"reported_epoch":499,"state":"active+undersized","last_fresh":"2023-10-11T11:19:00.636960+0000","last_change":"2023-10-11T11:15:31.679465+0000","last_active":"2023-10-11T11:19:00.636960+0000","last_peered":"2023-10-11T11:19:00.636960+0000","last_clean":"2023-10-11T11:15:27.638749+0000","last_became_active":"2023-10-11T11:15:31.679465+0000","last_became_peered":"2023-10-11T11:15:31.679465+0000","last_unstale":"2023-10-11T11:19:00.636960+0000","last_undegraded":"2023-10-11T11:19:00.636960+0000","last_fullsized":"2023-10-11T11:15:31.659135+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T16:03:42.656692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":11,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738810,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":785348,"kb_used_data":5812,"kb_used_omap":96,"kb_used_meta":779423,"kb_avail":93586492,"statfs":{"total":96636764160,"available":95832567808,"internally_reserved":0,"allocated":5951488,"data_stored":3106918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":98609,"internal_metadata":798129871},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 11:14:37 2023","interfaces":[{"interface":"back","average":{"1min":0.314,"5min":0.316,"15min":0.443},"min":{"1min":0.24399999999999999,"5min":0.14799999999999999,"15min":0.14599999999999999},"max":{"1min":0.48499999999999999,"5min":1.712,"15min":9.6400000000000006},"last":0.31},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.35499999999999998,"15min":0.41299999999999998},"min":{"1min":0.20899999999999999,"5min":0.17199999999999999,"15min":0.16400000000000001},"max":{"1min":3.8180000000000001,"5min":3.8180000000000001,"15min":6.3200000000000003},"last":0.26500000000000001}]},{"osd":1,"last update":"Wed Oct 11 11:19:40 2023","interfaces":[{"interface":"back","average":{"1min":0.29599999999999999,"5min":0.28499999999999998,"15min":0.32300000000000001},"min":{"1min":0.17699999999999999,"5min":0.154,"15min":0.154},"max":{"1min":0.40600000000000003,"5min":0.41999999999999998,"15min":2.9540000000000002},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.317,"5min":0.31,"15min":0.36199999999999999},"min":{"1min":0.20899999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.45000000000000001,"5min":0.49099999999999999,"15min":3.2719999999999998},"last":0.51900000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738756,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1484496,"kb_used_data":406792,"kb_used_omap":107,"kb_used_meta":1077588,"kb_avail":92887344,"statfs":{"total":96636764160,"available":95116640256,"internally_reserved":0,"allocated":416555008,"data_stored":413648921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110309,"internal_metadata":1103450395},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Oct 11 11:14:30 2023","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.35399999999999998,"15min":0.40600000000000003},"min":{"1min":0.18099999999999999,"5min":0.155,"15min":0.14899999999999999},"max":{"1min":1.4319999999999999,"5min":2.3460000000000001,"15min":6.593},"last":0.26200000000000001},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.39100000000000001,"15min":0.48199999999999998},"min":{"1min":0.189,"5min":0.189,"15min":0.183},"max":{"1min":2.2120000000000002,"5min":2.4089999999999998,"15min":6.5960000000000001},"last":0.316}]},{"osd":2,"last update":"Wed Oct 11 11:14:30 2023","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.35499999999999998,"15min":0.47099999999999997},"min":{"1min":0.20300000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":1.2490000000000001,"5min":2.665,"15min":8.4730000000000008},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.375,"15min":0.442},"min":{"1min":0.23200000000000001,"5min":0.20499999999999999,"15min":0.19600000000000001},"max":{"1min":1.28,"5min":2.6230000000000002,"15min":7.0129999999999999},"last":0.33600000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738811,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":826440,"kb_used_data":6392,"kb_used_omap":88,"kb_used_meta":819943,"kb_avail":93545400,"statfs":{"total":96636764160,"available":95790489600,"internally_reserved":0,"allocated":6545408,"data_stored":3697286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":90523,"internal_metadata":839622245},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 11:14:33 2023","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.42799999999999999,"15min":0.41099999999999998},"min":{"1min":0.151,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":1.2809999999999999,"5min":3.5739999999999998,"15min":8.2479999999999993},"last":0.223},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.48299999999999998,"15min":0.49199999999999999},"min":{"1min":0.20699999999999999,"5min":0.182,"15min":0.17999999999999999},"max":{"1min":1.323,"5min":3.6080000000000001,"15min":8.2870000000000008},"last":0.28399999999999997}]},{"osd":2,"last update":"Wed Oct 11 11:19:41 2023","interfaces":[{"interface":"back","average":{"1min":0.314,"5min":0.35399999999999998,"15min":0.35899999999999999},"min":{"1min":0.186,"5min":0.14699999999999999,"15min":0.14699999999999999},"max":{"1min":0.45500000000000002,"5min":7.4829999999999997,"15min":10.617000000000001},"last":0.30399999999999999},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.39400000000000002,"15min":0.35099999999999998},"min":{"1min":0.21199999999999999,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.49099999999999999,"5min":7.5350000000000001,"15min":7.5350000000000001},"last":0.34200000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-11T11:20:15.337 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-10-11T11:20:15.655 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T11:20:15.657 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2023-10-11T11:20:15.668 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":1598,"stamp":"2023-10-11T11:20:14.661430+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":11,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":107,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3096284,"kb_used_data":418996,"kb_used_omap":292,"kb_used_meta":2676955,"kb_avail":280019236,"statfs":{"total":289910292480,"available":286739697664,"internally_reserved":0,"allocated":429051904,"data_stored":420453125,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299441,"internal_metadata":2741202511},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003261"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":517,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-10-11T11:19:02.772828+0000","last_change":"2023-10-11T11:15:31.661747+0000","last_active":"2023-10-11T11:19:02.772828+0000","last_peered":"2023-10-11T11:19:02.772828+0000","last_clean":"2023-10-11T11:15:27.638215+0000","last_became_active":"2023-10-11T11:15:31.661747+0000","last_became_peered":"2023-10-11T11:15:31.661747+0000","last_unstale":"2023-10-11T11:19:02.772828+0000","last_undegraded":"2023-10-11T11:19:02.772828+0000","last_fullsized":"2023-10-11T11:15:31.649336+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:15:25.047039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":29,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-10-11T11:19:02.774213+0000","last_change":"2023-10-11T11:15:31.671363+0000","last_active":"2023-10-11T11:19:02.774213+0000","last_peered":"2023-10-11T11:19:02.774213+0000","last_clean":"2023-10-11T10:43:40.751300+0000","last_became_active":"2023-10-11T11:15:31.671363+0000","last_became_peered":"2023-10-11T11:15:31.671363+0000","last_unstale":"2023-10-11T11:19:02.774213+0000","last_undegraded":"2023-10-11T11:19:02.774213+0000","last_fullsized":"2023-10-11T11:15:31.652001+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:01:53.884964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":512,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-10-11T11:19:03.429640+0000","last_change":"2023-10-11T10:43:41.772781+0000","last_active":"2023-10-11T11:19:03.429640+0000","last_peered":"2023-10-11T11:19:03.429640+0000","last_clean":"2023-10-11T11:19:03.429640+0000","last_became_active":"2023-10-11T10:43:41.772443+0000","last_became_peered":"2023-10-11T10:43:41.772443+0000","last_unstale":"2023-10-11T11:19:03.429640+0000","last_undegraded":"2023-10-11T11:19:03.429640+0000","last_fullsized":"2023-10-11T11:19:03.429640+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T14:15:30.547284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":513,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-10-11T11:19:02.772211+0000","last_change":"2023-10-11T10:43:41.772615+0000","last_active":"2023-10-11T11:19:02.772211+0000","last_peered":"2023-10-11T11:19:02.772211+0000","last_clean":"2023-10-11T11:19:02.772211+0000","last_became_active":"2023-10-11T10:43:41.772290+0000","last_became_peered":"2023-10-11T10:43:41.772290+0000","last_unstale":"2023-10-11T11:19:02.772211+0000","last_undegraded":"2023-10-11T11:19:02.772211+0000","last_fullsized":"2023-10-11T11:19:02.772211+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:26:06.909963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":648,"reported_epoch":500,"state":"active+undersized+degraded","last_fresh":"2023-10-11T11:19:02.769176+0000","last_change":"2023-10-11T11:15:31.654022+0000","last_active":"2023-10-11T11:19:02.769176+0000","last_peered":"2023-10-11T11:19:02.769176+0000","last_clean":"2023-10-11T11:15:27.636884+0000","last_became_active":"2023-10-11T11:15:31.654022+0000","last_became_peered":"2023-10-11T11:15:31.654022+0000","last_unstale":"2023-10-11T11:19:02.769176+0000","last_undegraded":"2023-10-11T11:15:31.646946+0000","last_fullsized":"2023-10-11T11:15:31.646751+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:36.739637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:36.739637+0000","last_clean_scrub_stamp":"2023-10-11T10:43:36.739637+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T11:33:33.696447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":520,"reported_epoch":500,"state":"active+clean+remapped","last_fresh":"2023-10-11T11:19:03.429429+0000","last_change":"2023-10-11T11:15:33.660200+0000","last_active":"2023-10-11T11:19:03.429429+0000","last_peered":"2023-10-11T11:19:03.429429+0000","last_clean":"2023-10-11T11:19:03.429429+0000","last_became_active":"2023-10-11T11:15:33.659619+0000","last_became_peered":"2023-10-11T11:15:33.659619+0000","last_unstale":"2023-10-11T11:19:03.429429+0000","last_undegraded":"2023-10-11T11:19:03.429429+0000","last_fullsized":"2023-10-11T11:19:03.429429+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T17:21:16.866392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":517,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-10-11T11:19:02.775128+0000","last_change":"2023-10-11T11:15:31.670813+0000","last_active":"2023-10-11T11:19:02.775128+0000","last_peered":"2023-10-11T11:19:02.775128+0000","last_clean":"2023-10-11T11:15:27.638219+0000","last_became_active":"2023-10-11T11:15:31.670813+0000","last_became_peered":"2023-10-11T11:15:31.670813+0000","last_unstale":"2023-10-11T11:19:02.775128+0000","last_undegraded":"2023-10-11T11:19:02.775128+0000","last_fullsized":"2023-10-11T11:15:31.658785+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:41:50.872346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":516,"reported_epoch":499,"state":"active+undersized","last_fresh":"2023-10-11T11:19:00.631668+0000","last_change":"2023-10-11T11:15:31.666378+0000","last_active":"2023-10-11T11:19:00.631668+0000","last_peered":"2023-10-11T11:19:00.631668+0000","last_clean":"2023-10-11T11:15:27.638303+0000","last_became_active":"2023-10-11T11:15:31.666378+0000","last_became_peered":"2023-10-11T11:15:31.666378+0000","last_unstale":"2023-10-11T11:19:00.631668+0000","last_undegraded":"2023-10-11T11:19:00.631668+0000","last_fullsized":"2023-10-11T11:15:31.649186+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T15:35:19.331209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":516,"reported_epoch":499,"state":"active+undersized","last_fresh":"2023-10-11T11:19:00.636960+0000","last_change":"2023-10-11T11:15:31.679465+0000","last_active":"2023-10-11T11:19:00.636960+0000","last_peered":"2023-10-11T11:19:00.636960+0000","last_clean":"2023-10-11T11:15:27.638749+0000","last_became_active":"2023-10-11T11:15:31.679465+0000","last_became_peered":"2023-10-11T11:15:31.679465+0000","last_unstale":"2023-10-11T11:19:00.636960+0000","last_undegraded":"2023-10-11T11:19:00.636960+0000","last_fullsized":"2023-10-11T11:15:31.659135+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T16:03:42.656692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":11,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738810,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":785348,"kb_used_data":5812,"kb_used_omap":96,"kb_used_meta":779423,"kb_avail":93586492,"statfs":{"total":96636764160,"available":95832567808,"internally_reserved":0,"allocated":5951488,"data_stored":3106918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":98609,"internal_metadata":798129871},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 11:14:37 2023","interfaces":[{"interface":"back","average":{"1min":0.314,"5min":0.316,"15min":0.443},"min":{"1min":0.24399999999999999,"5min":0.14799999999999999,"15min":0.14599999999999999},"max":{"1min":0.48499999999999999,"5min":1.712,"15min":9.6400000000000006},"last":0.31},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.35499999999999998,"15min":0.41299999999999998},"min":{"1min":0.20899999999999999,"5min":0.17199999999999999,"15min":0.16400000000000001},"max":{"1min":3.8180000000000001,"5min":3.8180000000000001,"15min":6.3200000000000003},"last":0.26500000000000001}]},{"osd":1,"last update":"Wed Oct 11 11:19:40 2023","interfaces":[{"interface":"back","average":{"1min":0.29599999999999999,"5min":0.28499999999999998,"15min":0.32300000000000001},"min":{"1min":0.17699999999999999,"5min":0.154,"15min":0.154},"max":{"1min":0.40600000000000003,"5min":0.41999999999999998,"15min":2.9540000000000002},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.317,"5min":0.31,"15min":0.36199999999999999},"min":{"1min":0.20899999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.45000000000000001,"5min":0.49099999999999999,"15min":3.2719999999999998},"last":0.51900000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738756,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1484496,"kb_used_data":406792,"kb_used_omap":107,"kb_used_meta":1077588,"kb_avail":92887344,"statfs":{"total":96636764160,"available":95116640256,"internally_reserved":0,"allocated":416555008,"data_stored":413648921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110309,"internal_metadata":1103450395},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Oct 11 11:14:30 2023","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.35399999999999998,"15min":0.40600000000000003},"min":{"1min":0.18099999999999999,"5min":0.155,"15min":0.14899999999999999},"max":{"1min":1.4319999999999999,"5min":2.3460000000000001,"15min":6.593},"last":0.26200000000000001},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.39100000000000001,"15min":0.48199999999999998},"min":{"1min":0.189,"5min":0.189,"15min":0.183},"max":{"1min":2.2120000000000002,"5min":2.4089999999999998,"15min":6.5960000000000001},"last":0.316}]},{"osd":2,"last update":"Wed Oct 11 11:14:30 2023","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.35499999999999998,"15min":0.47099999999999997},"min":{"1min":0.20300000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":1.2490000000000001,"5min":2.665,"15min":8.4730000000000008},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.375,"15min":0.442},"min":{"1min":0.23200000000000001,"5min":0.20499999999999999,"15min":0.19600000000000001},"max":{"1min":1.28,"5min":2.6230000000000002,"15min":7.0129999999999999},"last":0.33600000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738811,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":826440,"kb_used_data":6392,"kb_used_omap":88,"kb_used_meta":819943,"kb_avail":93545400,"statfs":{"total":96636764160,"available":95790489600,"internally_reserved":0,"allocated":6545408,"data_stored":3697286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":90523,"internal_metadata":839622245},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 11:14:33 2023","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.42799999999999999,"15min":0.41099999999999998},"min":{"1min":0.151,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":1.2809999999999999,"5min":3.5739999999999998,"15min":8.2479999999999993},"last":0.223},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.48299999999999998,"15min":0.49199999999999999},"min":{"1min":0.20699999999999999,"5min":0.182,"15min":0.17999999999999999},"max":{"1min":1.323,"5min":3.6080000000000001,"15min":8.2870000000000008},"last":0.28399999999999997}]},{"osd":2,"last update":"Wed Oct 11 11:19:41 2023","interfaces":[{"interface":"back","average":{"1min":0.314,"5min":0.35399999999999998,"15min":0.35899999999999999},"min":{"1min":0.186,"5min":0.14699999999999999,"15min":0.14699999999999999},"max":{"1min":0.45500000000000002,"5min":7.4829999999999997,"15min":10.617000000000001},"last":0.30399999999999999},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.39400000000000002,"15min":0.35099999999999998},"min":{"1min":0.21199999999999999,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.49099999999999999,"5min":7.5350000000000001,"15min":7.5350000000000001},"last":0.34200000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-11T11:20:15.669 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-10-11T11:20:15.670 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 517, 'reported_epoch': 500, 'state': 'active+undersized', 'last_fresh': '2023-10-11T11:19:02.772828+0000', 'last_change': '2023-10-11T11:15:31.661747+0000', 'last_active': '2023-10-11T11:19:02.772828+0000', 'last_peered': '2023-10-11T11:19:02.772828+0000', 'last_clean': '2023-10-11T11:15:27.638215+0000', 'last_became_active': '2023-10-11T11:15:31.661747+0000', 'last_became_peered': '2023-10-11T11:15:31.661747+0000', 'last_unstale': '2023-10-11T11:19:02.772828+0000', 'last_undegraded': '2023-10-11T11:19:02.772828+0000', 'last_fullsized': '2023-10-11T11:15:31.649336+0000', 'mapping_epoch': 480, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_clean_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-12T22:15:25.047039+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-10-11T11:20:15.670 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-10-11T11:20:15.670 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 29, 'reported_epoch': 500, 'state': 'active+undersized', 'last_fresh': '2023-10-11T11:19:02.774213+0000', 'last_change': '2023-10-11T11:15:31.671363+0000', 'last_active': '2023-10-11T11:19:02.774213+0000', 'last_peered': '2023-10-11T11:19:02.774213+0000', 'last_clean': '2023-10-11T10:43:40.751300+0000', 'last_became_active': '2023-10-11T11:15:31.671363+0000', 'last_became_peered': '2023-10-11T11:15:31.671363+0000', 'last_unstale': '2023-10-11T11:19:02.774213+0000', 'last_undegraded': '2023-10-11T11:19:02.774213+0000', 'last_fullsized': '2023-10-11T11:15:31.652001+0000', 'mapping_epoch': 480, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_clean_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-12T21:01:53.884964+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-10-11T11:20:15.671 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-10-11T11:20:15.671 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "11'97", 'reported_seq': 648, 'reported_epoch': 500, 'state': 'active+undersized+degraded', 'last_fresh': '2023-10-11T11:19:02.769176+0000', 'last_change': '2023-10-11T11:15:31.654022+0000', 'last_active': '2023-10-11T11:19:02.769176+0000', 'last_peered': '2023-10-11T11:19:02.769176+0000', 'last_clean': '2023-10-11T11:15:27.636884+0000', 'last_became_active': '2023-10-11T11:15:31.654022+0000', 'last_became_peered': '2023-10-11T11:15:31.654022+0000', 'last_unstale': '2023-10-11T11:19:02.769176+0000', 'last_undegraded': '2023-10-11T11:15:31.646946+0000', 'last_fullsized': '2023-10-11T11:15:31.646751+0000', 'mapping_epoch': 480, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-11T10:43:36.739637+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-11T10:43:36.739637+0000', 'last_clean_scrub_stamp': '2023-10-11T10:43:36.739637+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-12T11:33:33.696447+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-10-11T11:20:15.671 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.0 is not active+clean 2023-10-11T11:20:15.672 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 520, 'reported_epoch': 500, 'state': 'active+clean+remapped', 'last_fresh': '2023-10-11T11:19:03.429429+0000', 'last_change': '2023-10-11T11:15:33.660200+0000', 'last_active': '2023-10-11T11:19:03.429429+0000', 'last_peered': '2023-10-11T11:19:03.429429+0000', 'last_clean': '2023-10-11T11:19:03.429429+0000', 'last_became_active': '2023-10-11T11:15:33.659619+0000', 'last_became_peered': '2023-10-11T11:15:33.659619+0000', 'last_unstale': '2023-10-11T11:19:03.429429+0000', 'last_undegraded': '2023-10-11T11:19:03.429429+0000', 'last_fullsized': '2023-10-11T11:19:03.429429+0000', 'mapping_epoch': 482, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 483, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_clean_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-12T17:21:16.866392+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2, 1], 'avail_no_missing': ['2', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-10-11T11:20:15.672 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-10-11T11:20:15.672 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 517, 'reported_epoch': 500, 'state': 'active+undersized', 'last_fresh': '2023-10-11T11:19:02.775128+0000', 'last_change': '2023-10-11T11:15:31.670813+0000', 'last_active': '2023-10-11T11:19:02.775128+0000', 'last_peered': '2023-10-11T11:19:02.775128+0000', 'last_clean': '2023-10-11T11:15:27.638219+0000', 'last_became_active': '2023-10-11T11:15:31.670813+0000', 'last_became_peered': '2023-10-11T11:15:31.670813+0000', 'last_unstale': '2023-10-11T11:19:02.775128+0000', 'last_undegraded': '2023-10-11T11:19:02.775128+0000', 'last_fullsized': '2023-10-11T11:15:31.658785+0000', 'mapping_epoch': 480, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_clean_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-12T21:41:50.872346+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-10-11T11:20:15.673 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-10-11T11:20:15.673 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 516, 'reported_epoch': 499, 'state': 'active+undersized', 'last_fresh': '2023-10-11T11:19:00.631668+0000', 'last_change': '2023-10-11T11:15:31.666378+0000', 'last_active': '2023-10-11T11:19:00.631668+0000', 'last_peered': '2023-10-11T11:19:00.631668+0000', 'last_clean': '2023-10-11T11:15:27.638303+0000', 'last_became_active': '2023-10-11T11:15:31.666378+0000', 'last_became_peered': '2023-10-11T11:15:31.666378+0000', 'last_unstale': '2023-10-11T11:19:00.631668+0000', 'last_undegraded': '2023-10-11T11:19:00.631668+0000', 'last_fullsized': '2023-10-11T11:15:31.649186+0000', 'mapping_epoch': 480, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_clean_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-12T15:35:19.331209+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-10-11T11:20:15.673 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-10-11T11:20:15.673 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 516, 'reported_epoch': 499, 'state': 'active+undersized', 'last_fresh': '2023-10-11T11:19:00.636960+0000', 'last_change': '2023-10-11T11:15:31.679465+0000', 'last_active': '2023-10-11T11:19:00.636960+0000', 'last_peered': '2023-10-11T11:19:00.636960+0000', 'last_clean': '2023-10-11T11:15:27.638749+0000', 'last_became_active': '2023-10-11T11:15:31.679465+0000', 'last_became_peered': '2023-10-11T11:15:31.679465+0000', 'last_unstale': '2023-10-11T11:19:00.636960+0000', 'last_undegraded': '2023-10-11T11:19:00.636960+0000', 'last_fullsized': '2023-10-11T11:15:31.659135+0000', 'mapping_epoch': 480, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_clean_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-12T16:03:42.656692+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-10-11T11:20:15.674 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-10-11T11:20:16.069 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T11:20:16.069 INFO:teuthology.orchestra.run.smithi111.stdout:{"fsid":"022ce6f2-9a5b-45ad-8b77-a1332d92f228","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2211,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":500,"num_osds":3,"num_up_osds":2,"osd_up_since":1697022930,"num_in_osds":3,"osd_in_since":1697021005,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":2},{"state_name":"active+undersized+degraded","count":1},{"state_name":"active+clean+remapped","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3170594816,"bytes_avail":286739697664,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":36,"modified":"2023-10-11T11:19:04.644408+0000","services":{}},"progress_events":{}} 2023-10-11T11:20:16.083 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-10-11T11:20:16.083 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-10-11T11:20:16.397 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T11:20:16.399 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2023-10-11T11:20:16.410 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":1598,"stamp":"2023-10-11T11:20:14.661430+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":11,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":107,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3096284,"kb_used_data":418996,"kb_used_omap":292,"kb_used_meta":2676955,"kb_avail":280019236,"statfs":{"total":289910292480,"available":286739697664,"internally_reserved":0,"allocated":429051904,"data_stored":420453125,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299441,"internal_metadata":2741202511},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003261"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":517,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-10-11T11:19:02.772828+0000","last_change":"2023-10-11T11:15:31.661747+0000","last_active":"2023-10-11T11:19:02.772828+0000","last_peered":"2023-10-11T11:19:02.772828+0000","last_clean":"2023-10-11T11:15:27.638215+0000","last_became_active":"2023-10-11T11:15:31.661747+0000","last_became_peered":"2023-10-11T11:15:31.661747+0000","last_unstale":"2023-10-11T11:19:02.772828+0000","last_undegraded":"2023-10-11T11:19:02.772828+0000","last_fullsized":"2023-10-11T11:15:31.649336+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:15:25.047039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":29,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-10-11T11:19:02.774213+0000","last_change":"2023-10-11T11:15:31.671363+0000","last_active":"2023-10-11T11:19:02.774213+0000","last_peered":"2023-10-11T11:19:02.774213+0000","last_clean":"2023-10-11T10:43:40.751300+0000","last_became_active":"2023-10-11T11:15:31.671363+0000","last_became_peered":"2023-10-11T11:15:31.671363+0000","last_unstale":"2023-10-11T11:19:02.774213+0000","last_undegraded":"2023-10-11T11:19:02.774213+0000","last_fullsized":"2023-10-11T11:15:31.652001+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:01:53.884964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":512,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-10-11T11:19:03.429640+0000","last_change":"2023-10-11T10:43:41.772781+0000","last_active":"2023-10-11T11:19:03.429640+0000","last_peered":"2023-10-11T11:19:03.429640+0000","last_clean":"2023-10-11T11:19:03.429640+0000","last_became_active":"2023-10-11T10:43:41.772443+0000","last_became_peered":"2023-10-11T10:43:41.772443+0000","last_unstale":"2023-10-11T11:19:03.429640+0000","last_undegraded":"2023-10-11T11:19:03.429640+0000","last_fullsized":"2023-10-11T11:19:03.429640+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T14:15:30.547284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":513,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-10-11T11:19:02.772211+0000","last_change":"2023-10-11T10:43:41.772615+0000","last_active":"2023-10-11T11:19:02.772211+0000","last_peered":"2023-10-11T11:19:02.772211+0000","last_clean":"2023-10-11T11:19:02.772211+0000","last_became_active":"2023-10-11T10:43:41.772290+0000","last_became_peered":"2023-10-11T10:43:41.772290+0000","last_unstale":"2023-10-11T11:19:02.772211+0000","last_undegraded":"2023-10-11T11:19:02.772211+0000","last_fullsized":"2023-10-11T11:19:02.772211+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:26:06.909963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":648,"reported_epoch":500,"state":"active+undersized+degraded","last_fresh":"2023-10-11T11:19:02.769176+0000","last_change":"2023-10-11T11:15:31.654022+0000","last_active":"2023-10-11T11:19:02.769176+0000","last_peered":"2023-10-11T11:19:02.769176+0000","last_clean":"2023-10-11T11:15:27.636884+0000","last_became_active":"2023-10-11T11:15:31.654022+0000","last_became_peered":"2023-10-11T11:15:31.654022+0000","last_unstale":"2023-10-11T11:19:02.769176+0000","last_undegraded":"2023-10-11T11:15:31.646946+0000","last_fullsized":"2023-10-11T11:15:31.646751+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:36.739637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:36.739637+0000","last_clean_scrub_stamp":"2023-10-11T10:43:36.739637+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T11:33:33.696447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":520,"reported_epoch":500,"state":"active+clean+remapped","last_fresh":"2023-10-11T11:19:03.429429+0000","last_change":"2023-10-11T11:15:33.660200+0000","last_active":"2023-10-11T11:19:03.429429+0000","last_peered":"2023-10-11T11:19:03.429429+0000","last_clean":"2023-10-11T11:19:03.429429+0000","last_became_active":"2023-10-11T11:15:33.659619+0000","last_became_peered":"2023-10-11T11:15:33.659619+0000","last_unstale":"2023-10-11T11:19:03.429429+0000","last_undegraded":"2023-10-11T11:19:03.429429+0000","last_fullsized":"2023-10-11T11:19:03.429429+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T17:21:16.866392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":517,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-10-11T11:19:02.775128+0000","last_change":"2023-10-11T11:15:31.670813+0000","last_active":"2023-10-11T11:19:02.775128+0000","last_peered":"2023-10-11T11:19:02.775128+0000","last_clean":"2023-10-11T11:15:27.638219+0000","last_became_active":"2023-10-11T11:15:31.670813+0000","last_became_peered":"2023-10-11T11:15:31.670813+0000","last_unstale":"2023-10-11T11:19:02.775128+0000","last_undegraded":"2023-10-11T11:19:02.775128+0000","last_fullsized":"2023-10-11T11:15:31.658785+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:41:50.872346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":516,"reported_epoch":499,"state":"active+undersized","last_fresh":"2023-10-11T11:19:00.631668+0000","last_change":"2023-10-11T11:15:31.666378+0000","last_active":"2023-10-11T11:19:00.631668+0000","last_peered":"2023-10-11T11:19:00.631668+0000","last_clean":"2023-10-11T11:15:27.638303+0000","last_became_active":"2023-10-11T11:15:31.666378+0000","last_became_peered":"2023-10-11T11:15:31.666378+0000","last_unstale":"2023-10-11T11:19:00.631668+0000","last_undegraded":"2023-10-11T11:19:00.631668+0000","last_fullsized":"2023-10-11T11:15:31.649186+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T15:35:19.331209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":516,"reported_epoch":499,"state":"active+undersized","last_fresh":"2023-10-11T11:19:00.636960+0000","last_change":"2023-10-11T11:15:31.679465+0000","last_active":"2023-10-11T11:19:00.636960+0000","last_peered":"2023-10-11T11:19:00.636960+0000","last_clean":"2023-10-11T11:15:27.638749+0000","last_became_active":"2023-10-11T11:15:31.679465+0000","last_became_peered":"2023-10-11T11:15:31.679465+0000","last_unstale":"2023-10-11T11:19:00.636960+0000","last_undegraded":"2023-10-11T11:19:00.636960+0000","last_fullsized":"2023-10-11T11:15:31.659135+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T16:03:42.656692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":11,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738810,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":785348,"kb_used_data":5812,"kb_used_omap":96,"kb_used_meta":779423,"kb_avail":93586492,"statfs":{"total":96636764160,"available":95832567808,"internally_reserved":0,"allocated":5951488,"data_stored":3106918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":98609,"internal_metadata":798129871},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 11:14:37 2023","interfaces":[{"interface":"back","average":{"1min":0.314,"5min":0.316,"15min":0.443},"min":{"1min":0.24399999999999999,"5min":0.14799999999999999,"15min":0.14599999999999999},"max":{"1min":0.48499999999999999,"5min":1.712,"15min":9.6400000000000006},"last":0.31},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.35499999999999998,"15min":0.41299999999999998},"min":{"1min":0.20899999999999999,"5min":0.17199999999999999,"15min":0.16400000000000001},"max":{"1min":3.8180000000000001,"5min":3.8180000000000001,"15min":6.3200000000000003},"last":0.26500000000000001}]},{"osd":1,"last update":"Wed Oct 11 11:19:40 2023","interfaces":[{"interface":"back","average":{"1min":0.29599999999999999,"5min":0.28499999999999998,"15min":0.32300000000000001},"min":{"1min":0.17699999999999999,"5min":0.154,"15min":0.154},"max":{"1min":0.40600000000000003,"5min":0.41999999999999998,"15min":2.9540000000000002},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.317,"5min":0.31,"15min":0.36199999999999999},"min":{"1min":0.20899999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.45000000000000001,"5min":0.49099999999999999,"15min":3.2719999999999998},"last":0.51900000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738756,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1484496,"kb_used_data":406792,"kb_used_omap":107,"kb_used_meta":1077588,"kb_avail":92887344,"statfs":{"total":96636764160,"available":95116640256,"internally_reserved":0,"allocated":416555008,"data_stored":413648921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110309,"internal_metadata":1103450395},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Oct 11 11:14:30 2023","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.35399999999999998,"15min":0.40600000000000003},"min":{"1min":0.18099999999999999,"5min":0.155,"15min":0.14899999999999999},"max":{"1min":1.4319999999999999,"5min":2.3460000000000001,"15min":6.593},"last":0.26200000000000001},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.39100000000000001,"15min":0.48199999999999998},"min":{"1min":0.189,"5min":0.189,"15min":0.183},"max":{"1min":2.2120000000000002,"5min":2.4089999999999998,"15min":6.5960000000000001},"last":0.316}]},{"osd":2,"last update":"Wed Oct 11 11:14:30 2023","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.35499999999999998,"15min":0.47099999999999997},"min":{"1min":0.20300000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":1.2490000000000001,"5min":2.665,"15min":8.4730000000000008},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.375,"15min":0.442},"min":{"1min":0.23200000000000001,"5min":0.20499999999999999,"15min":0.19600000000000001},"max":{"1min":1.28,"5min":2.6230000000000002,"15min":7.0129999999999999},"last":0.33600000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738811,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":826440,"kb_used_data":6392,"kb_used_omap":88,"kb_used_meta":819943,"kb_avail":93545400,"statfs":{"total":96636764160,"available":95790489600,"internally_reserved":0,"allocated":6545408,"data_stored":3697286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":90523,"internal_metadata":839622245},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 11:14:33 2023","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.42799999999999999,"15min":0.41099999999999998},"min":{"1min":0.151,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":1.2809999999999999,"5min":3.5739999999999998,"15min":8.2479999999999993},"last":0.223},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.48299999999999998,"15min":0.49199999999999999},"min":{"1min":0.20699999999999999,"5min":0.182,"15min":0.17999999999999999},"max":{"1min":1.323,"5min":3.6080000000000001,"15min":8.2870000000000008},"last":0.28399999999999997}]},{"osd":2,"last update":"Wed Oct 11 11:19:41 2023","interfaces":[{"interface":"back","average":{"1min":0.314,"5min":0.35399999999999998,"15min":0.35899999999999999},"min":{"1min":0.186,"5min":0.14699999999999999,"15min":0.14699999999999999},"max":{"1min":0.45500000000000002,"5min":7.4829999999999997,"15min":10.617000000000001},"last":0.30399999999999999},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.39400000000000002,"15min":0.35099999999999998},"min":{"1min":0.21199999999999999,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.49099999999999999,"5min":7.5350000000000001,"15min":7.5350000000000001},"last":0.34200000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-11T11:20:18.562 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~286s 2023-10-11T11:20:19.412 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-10-11T11:20:19.740 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T11:20:19.742 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2023-10-11T11:20:19.754 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":1600,"stamp":"2023-10-11T11:20:18.662341+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":11,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":107,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3096284,"kb_used_data":418996,"kb_used_omap":292,"kb_used_meta":2676955,"kb_avail":280019236,"statfs":{"total":289910292480,"available":286739697664,"internally_reserved":0,"allocated":429051904,"data_stored":420453125,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299441,"internal_metadata":2741202511},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002970"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":517,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-10-11T11:19:02.772828+0000","last_change":"2023-10-11T11:15:31.661747+0000","last_active":"2023-10-11T11:19:02.772828+0000","last_peered":"2023-10-11T11:19:02.772828+0000","last_clean":"2023-10-11T11:15:27.638215+0000","last_became_active":"2023-10-11T11:15:31.661747+0000","last_became_peered":"2023-10-11T11:15:31.661747+0000","last_unstale":"2023-10-11T11:19:02.772828+0000","last_undegraded":"2023-10-11T11:19:02.772828+0000","last_fullsized":"2023-10-11T11:15:31.649336+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:15:25.047039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":29,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-10-11T11:19:02.774213+0000","last_change":"2023-10-11T11:15:31.671363+0000","last_active":"2023-10-11T11:19:02.774213+0000","last_peered":"2023-10-11T11:19:02.774213+0000","last_clean":"2023-10-11T10:43:40.751300+0000","last_became_active":"2023-10-11T11:15:31.671363+0000","last_became_peered":"2023-10-11T11:15:31.671363+0000","last_unstale":"2023-10-11T11:19:02.774213+0000","last_undegraded":"2023-10-11T11:19:02.774213+0000","last_fullsized":"2023-10-11T11:15:31.652001+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:01:53.884964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":512,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-10-11T11:19:03.429640+0000","last_change":"2023-10-11T10:43:41.772781+0000","last_active":"2023-10-11T11:19:03.429640+0000","last_peered":"2023-10-11T11:19:03.429640+0000","last_clean":"2023-10-11T11:19:03.429640+0000","last_became_active":"2023-10-11T10:43:41.772443+0000","last_became_peered":"2023-10-11T10:43:41.772443+0000","last_unstale":"2023-10-11T11:19:03.429640+0000","last_undegraded":"2023-10-11T11:19:03.429640+0000","last_fullsized":"2023-10-11T11:19:03.429640+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T14:15:30.547284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":513,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-10-11T11:19:02.772211+0000","last_change":"2023-10-11T10:43:41.772615+0000","last_active":"2023-10-11T11:19:02.772211+0000","last_peered":"2023-10-11T11:19:02.772211+0000","last_clean":"2023-10-11T11:19:02.772211+0000","last_became_active":"2023-10-11T10:43:41.772290+0000","last_became_peered":"2023-10-11T10:43:41.772290+0000","last_unstale":"2023-10-11T11:19:02.772211+0000","last_undegraded":"2023-10-11T11:19:02.772211+0000","last_fullsized":"2023-10-11T11:19:02.772211+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:26:06.909963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":648,"reported_epoch":500,"state":"active+undersized+degraded","last_fresh":"2023-10-11T11:19:02.769176+0000","last_change":"2023-10-11T11:15:31.654022+0000","last_active":"2023-10-11T11:19:02.769176+0000","last_peered":"2023-10-11T11:19:02.769176+0000","last_clean":"2023-10-11T11:15:27.636884+0000","last_became_active":"2023-10-11T11:15:31.654022+0000","last_became_peered":"2023-10-11T11:15:31.654022+0000","last_unstale":"2023-10-11T11:19:02.769176+0000","last_undegraded":"2023-10-11T11:15:31.646946+0000","last_fullsized":"2023-10-11T11:15:31.646751+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:36.739637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:36.739637+0000","last_clean_scrub_stamp":"2023-10-11T10:43:36.739637+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T11:33:33.696447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":520,"reported_epoch":500,"state":"active+clean+remapped","last_fresh":"2023-10-11T11:19:03.429429+0000","last_change":"2023-10-11T11:15:33.660200+0000","last_active":"2023-10-11T11:19:03.429429+0000","last_peered":"2023-10-11T11:19:03.429429+0000","last_clean":"2023-10-11T11:19:03.429429+0000","last_became_active":"2023-10-11T11:15:33.659619+0000","last_became_peered":"2023-10-11T11:15:33.659619+0000","last_unstale":"2023-10-11T11:19:03.429429+0000","last_undegraded":"2023-10-11T11:19:03.429429+0000","last_fullsized":"2023-10-11T11:19:03.429429+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T17:21:16.866392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":517,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-10-11T11:19:02.775128+0000","last_change":"2023-10-11T11:15:31.670813+0000","last_active":"2023-10-11T11:19:02.775128+0000","last_peered":"2023-10-11T11:19:02.775128+0000","last_clean":"2023-10-11T11:15:27.638219+0000","last_became_active":"2023-10-11T11:15:31.670813+0000","last_became_peered":"2023-10-11T11:15:31.670813+0000","last_unstale":"2023-10-11T11:19:02.775128+0000","last_undegraded":"2023-10-11T11:19:02.775128+0000","last_fullsized":"2023-10-11T11:15:31.658785+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:41:50.872346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":516,"reported_epoch":499,"state":"active+undersized","last_fresh":"2023-10-11T11:19:00.631668+0000","last_change":"2023-10-11T11:15:31.666378+0000","last_active":"2023-10-11T11:19:00.631668+0000","last_peered":"2023-10-11T11:19:00.631668+0000","last_clean":"2023-10-11T11:15:27.638303+0000","last_became_active":"2023-10-11T11:15:31.666378+0000","last_became_peered":"2023-10-11T11:15:31.666378+0000","last_unstale":"2023-10-11T11:19:00.631668+0000","last_undegraded":"2023-10-11T11:19:00.631668+0000","last_fullsized":"2023-10-11T11:15:31.649186+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T15:35:19.331209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":516,"reported_epoch":499,"state":"active+undersized","last_fresh":"2023-10-11T11:19:00.636960+0000","last_change":"2023-10-11T11:15:31.679465+0000","last_active":"2023-10-11T11:19:00.636960+0000","last_peered":"2023-10-11T11:19:00.636960+0000","last_clean":"2023-10-11T11:15:27.638749+0000","last_became_active":"2023-10-11T11:15:31.679465+0000","last_became_peered":"2023-10-11T11:15:31.679465+0000","last_unstale":"2023-10-11T11:19:00.636960+0000","last_undegraded":"2023-10-11T11:19:00.636960+0000","last_fullsized":"2023-10-11T11:15:31.659135+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T16:03:42.656692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":11,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738811,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":785348,"kb_used_data":5812,"kb_used_omap":96,"kb_used_meta":779423,"kb_avail":93586492,"statfs":{"total":96636764160,"available":95832567808,"internally_reserved":0,"allocated":5951488,"data_stored":3106918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":98609,"internal_metadata":798129871},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 11:14:37 2023","interfaces":[{"interface":"back","average":{"1min":0.314,"5min":0.316,"15min":0.443},"min":{"1min":0.24399999999999999,"5min":0.14799999999999999,"15min":0.14599999999999999},"max":{"1min":0.48499999999999999,"5min":1.712,"15min":9.6400000000000006},"last":0.31},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.35499999999999998,"15min":0.41299999999999998},"min":{"1min":0.20899999999999999,"5min":0.17199999999999999,"15min":0.16400000000000001},"max":{"1min":3.8180000000000001,"5min":3.8180000000000001,"15min":6.3200000000000003},"last":0.26500000000000001}]},{"osd":1,"last update":"Wed Oct 11 11:19:40 2023","interfaces":[{"interface":"back","average":{"1min":0.29599999999999999,"5min":0.28499999999999998,"15min":0.32300000000000001},"min":{"1min":0.17699999999999999,"5min":0.154,"15min":0.154},"max":{"1min":0.40600000000000003,"5min":0.41999999999999998,"15min":2.9540000000000002},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.317,"5min":0.31,"15min":0.36199999999999999},"min":{"1min":0.20899999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.45000000000000001,"5min":0.49099999999999999,"15min":3.2719999999999998},"last":0.51900000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738756,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1484496,"kb_used_data":406792,"kb_used_omap":107,"kb_used_meta":1077588,"kb_avail":92887344,"statfs":{"total":96636764160,"available":95116640256,"internally_reserved":0,"allocated":416555008,"data_stored":413648921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110309,"internal_metadata":1103450395},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Oct 11 11:14:30 2023","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.35399999999999998,"15min":0.40600000000000003},"min":{"1min":0.18099999999999999,"5min":0.155,"15min":0.14899999999999999},"max":{"1min":1.4319999999999999,"5min":2.3460000000000001,"15min":6.593},"last":0.26200000000000001},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.39100000000000001,"15min":0.48199999999999998},"min":{"1min":0.189,"5min":0.189,"15min":0.183},"max":{"1min":2.2120000000000002,"5min":2.4089999999999998,"15min":6.5960000000000001},"last":0.316}]},{"osd":2,"last update":"Wed Oct 11 11:14:30 2023","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.35499999999999998,"15min":0.47099999999999997},"min":{"1min":0.20300000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":1.2490000000000001,"5min":2.665,"15min":8.4730000000000008},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.375,"15min":0.442},"min":{"1min":0.23200000000000001,"5min":0.20499999999999999,"15min":0.19600000000000001},"max":{"1min":1.28,"5min":2.6230000000000002,"15min":7.0129999999999999},"last":0.33600000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738812,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":826440,"kb_used_data":6392,"kb_used_omap":88,"kb_used_meta":819943,"kb_avail":93545400,"statfs":{"total":96636764160,"available":95790489600,"internally_reserved":0,"allocated":6545408,"data_stored":3697286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":90523,"internal_metadata":839622245},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 11:14:33 2023","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.42799999999999999,"15min":0.41099999999999998},"min":{"1min":0.151,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":1.2809999999999999,"5min":3.5739999999999998,"15min":8.2479999999999993},"last":0.223},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.48299999999999998,"15min":0.49199999999999999},"min":{"1min":0.20699999999999999,"5min":0.182,"15min":0.17999999999999999},"max":{"1min":1.323,"5min":3.6080000000000001,"15min":8.2870000000000008},"last":0.28399999999999997}]},{"osd":2,"last update":"Wed Oct 11 11:19:41 2023","interfaces":[{"interface":"back","average":{"1min":0.314,"5min":0.35399999999999998,"15min":0.35899999999999999},"min":{"1min":0.186,"5min":0.14699999999999999,"15min":0.14699999999999999},"max":{"1min":0.45500000000000002,"5min":7.4829999999999997,"15min":10.617000000000001},"last":0.28100000000000003},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.39400000000000002,"15min":0.35099999999999998},"min":{"1min":0.21199999999999999,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.49099999999999999,"5min":7.5350000000000001,"15min":7.5350000000000001},"last":0.317}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-11T11:20:19.755 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-10-11T11:20:20.071 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T11:20:20.074 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2023-10-11T11:20:20.085 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":1600,"stamp":"2023-10-11T11:20:18.662341+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":11,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":107,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3096284,"kb_used_data":418996,"kb_used_omap":292,"kb_used_meta":2676955,"kb_avail":280019236,"statfs":{"total":289910292480,"available":286739697664,"internally_reserved":0,"allocated":429051904,"data_stored":420453125,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299441,"internal_metadata":2741202511},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002970"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":517,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-10-11T11:19:02.772828+0000","last_change":"2023-10-11T11:15:31.661747+0000","last_active":"2023-10-11T11:19:02.772828+0000","last_peered":"2023-10-11T11:19:02.772828+0000","last_clean":"2023-10-11T11:15:27.638215+0000","last_became_active":"2023-10-11T11:15:31.661747+0000","last_became_peered":"2023-10-11T11:15:31.661747+0000","last_unstale":"2023-10-11T11:19:02.772828+0000","last_undegraded":"2023-10-11T11:19:02.772828+0000","last_fullsized":"2023-10-11T11:15:31.649336+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:15:25.047039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":29,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-10-11T11:19:02.774213+0000","last_change":"2023-10-11T11:15:31.671363+0000","last_active":"2023-10-11T11:19:02.774213+0000","last_peered":"2023-10-11T11:19:02.774213+0000","last_clean":"2023-10-11T10:43:40.751300+0000","last_became_active":"2023-10-11T11:15:31.671363+0000","last_became_peered":"2023-10-11T11:15:31.671363+0000","last_unstale":"2023-10-11T11:19:02.774213+0000","last_undegraded":"2023-10-11T11:19:02.774213+0000","last_fullsized":"2023-10-11T11:15:31.652001+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:01:53.884964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":512,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-10-11T11:19:03.429640+0000","last_change":"2023-10-11T10:43:41.772781+0000","last_active":"2023-10-11T11:19:03.429640+0000","last_peered":"2023-10-11T11:19:03.429640+0000","last_clean":"2023-10-11T11:19:03.429640+0000","last_became_active":"2023-10-11T10:43:41.772443+0000","last_became_peered":"2023-10-11T10:43:41.772443+0000","last_unstale":"2023-10-11T11:19:03.429640+0000","last_undegraded":"2023-10-11T11:19:03.429640+0000","last_fullsized":"2023-10-11T11:19:03.429640+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T14:15:30.547284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":513,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-10-11T11:19:02.772211+0000","last_change":"2023-10-11T10:43:41.772615+0000","last_active":"2023-10-11T11:19:02.772211+0000","last_peered":"2023-10-11T11:19:02.772211+0000","last_clean":"2023-10-11T11:19:02.772211+0000","last_became_active":"2023-10-11T10:43:41.772290+0000","last_became_peered":"2023-10-11T10:43:41.772290+0000","last_unstale":"2023-10-11T11:19:02.772211+0000","last_undegraded":"2023-10-11T11:19:02.772211+0000","last_fullsized":"2023-10-11T11:19:02.772211+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:26:06.909963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":648,"reported_epoch":500,"state":"active+undersized+degraded","last_fresh":"2023-10-11T11:19:02.769176+0000","last_change":"2023-10-11T11:15:31.654022+0000","last_active":"2023-10-11T11:19:02.769176+0000","last_peered":"2023-10-11T11:19:02.769176+0000","last_clean":"2023-10-11T11:15:27.636884+0000","last_became_active":"2023-10-11T11:15:31.654022+0000","last_became_peered":"2023-10-11T11:15:31.654022+0000","last_unstale":"2023-10-11T11:19:02.769176+0000","last_undegraded":"2023-10-11T11:15:31.646946+0000","last_fullsized":"2023-10-11T11:15:31.646751+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:36.739637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:36.739637+0000","last_clean_scrub_stamp":"2023-10-11T10:43:36.739637+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T11:33:33.696447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":520,"reported_epoch":500,"state":"active+clean+remapped","last_fresh":"2023-10-11T11:19:03.429429+0000","last_change":"2023-10-11T11:15:33.660200+0000","last_active":"2023-10-11T11:19:03.429429+0000","last_peered":"2023-10-11T11:19:03.429429+0000","last_clean":"2023-10-11T11:19:03.429429+0000","last_became_active":"2023-10-11T11:15:33.659619+0000","last_became_peered":"2023-10-11T11:15:33.659619+0000","last_unstale":"2023-10-11T11:19:03.429429+0000","last_undegraded":"2023-10-11T11:19:03.429429+0000","last_fullsized":"2023-10-11T11:19:03.429429+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T17:21:16.866392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":517,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-10-11T11:19:02.775128+0000","last_change":"2023-10-11T11:15:31.670813+0000","last_active":"2023-10-11T11:19:02.775128+0000","last_peered":"2023-10-11T11:19:02.775128+0000","last_clean":"2023-10-11T11:15:27.638219+0000","last_became_active":"2023-10-11T11:15:31.670813+0000","last_became_peered":"2023-10-11T11:15:31.670813+0000","last_unstale":"2023-10-11T11:19:02.775128+0000","last_undegraded":"2023-10-11T11:19:02.775128+0000","last_fullsized":"2023-10-11T11:15:31.658785+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:41:50.872346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":516,"reported_epoch":499,"state":"active+undersized","last_fresh":"2023-10-11T11:19:00.631668+0000","last_change":"2023-10-11T11:15:31.666378+0000","last_active":"2023-10-11T11:19:00.631668+0000","last_peered":"2023-10-11T11:19:00.631668+0000","last_clean":"2023-10-11T11:15:27.638303+0000","last_became_active":"2023-10-11T11:15:31.666378+0000","last_became_peered":"2023-10-11T11:15:31.666378+0000","last_unstale":"2023-10-11T11:19:00.631668+0000","last_undegraded":"2023-10-11T11:19:00.631668+0000","last_fullsized":"2023-10-11T11:15:31.649186+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T15:35:19.331209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":516,"reported_epoch":499,"state":"active+undersized","last_fresh":"2023-10-11T11:19:00.636960+0000","last_change":"2023-10-11T11:15:31.679465+0000","last_active":"2023-10-11T11:19:00.636960+0000","last_peered":"2023-10-11T11:19:00.636960+0000","last_clean":"2023-10-11T11:15:27.638749+0000","last_became_active":"2023-10-11T11:15:31.679465+0000","last_became_peered":"2023-10-11T11:15:31.679465+0000","last_unstale":"2023-10-11T11:19:00.636960+0000","last_undegraded":"2023-10-11T11:19:00.636960+0000","last_fullsized":"2023-10-11T11:15:31.659135+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T16:03:42.656692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":11,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738811,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":785348,"kb_used_data":5812,"kb_used_omap":96,"kb_used_meta":779423,"kb_avail":93586492,"statfs":{"total":96636764160,"available":95832567808,"internally_reserved":0,"allocated":5951488,"data_stored":3106918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":98609,"internal_metadata":798129871},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 11:14:37 2023","interfaces":[{"interface":"back","average":{"1min":0.314,"5min":0.316,"15min":0.443},"min":{"1min":0.24399999999999999,"5min":0.14799999999999999,"15min":0.14599999999999999},"max":{"1min":0.48499999999999999,"5min":1.712,"15min":9.6400000000000006},"last":0.31},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.35499999999999998,"15min":0.41299999999999998},"min":{"1min":0.20899999999999999,"5min":0.17199999999999999,"15min":0.16400000000000001},"max":{"1min":3.8180000000000001,"5min":3.8180000000000001,"15min":6.3200000000000003},"last":0.26500000000000001}]},{"osd":1,"last update":"Wed Oct 11 11:19:40 2023","interfaces":[{"interface":"back","average":{"1min":0.29599999999999999,"5min":0.28499999999999998,"15min":0.32300000000000001},"min":{"1min":0.17699999999999999,"5min":0.154,"15min":0.154},"max":{"1min":0.40600000000000003,"5min":0.41999999999999998,"15min":2.9540000000000002},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.317,"5min":0.31,"15min":0.36199999999999999},"min":{"1min":0.20899999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.45000000000000001,"5min":0.49099999999999999,"15min":3.2719999999999998},"last":0.51900000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738756,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1484496,"kb_used_data":406792,"kb_used_omap":107,"kb_used_meta":1077588,"kb_avail":92887344,"statfs":{"total":96636764160,"available":95116640256,"internally_reserved":0,"allocated":416555008,"data_stored":413648921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110309,"internal_metadata":1103450395},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Oct 11 11:14:30 2023","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.35399999999999998,"15min":0.40600000000000003},"min":{"1min":0.18099999999999999,"5min":0.155,"15min":0.14899999999999999},"max":{"1min":1.4319999999999999,"5min":2.3460000000000001,"15min":6.593},"last":0.26200000000000001},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.39100000000000001,"15min":0.48199999999999998},"min":{"1min":0.189,"5min":0.189,"15min":0.183},"max":{"1min":2.2120000000000002,"5min":2.4089999999999998,"15min":6.5960000000000001},"last":0.316}]},{"osd":2,"last update":"Wed Oct 11 11:14:30 2023","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.35499999999999998,"15min":0.47099999999999997},"min":{"1min":0.20300000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":1.2490000000000001,"5min":2.665,"15min":8.4730000000000008},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.375,"15min":0.442},"min":{"1min":0.23200000000000001,"5min":0.20499999999999999,"15min":0.19600000000000001},"max":{"1min":1.28,"5min":2.6230000000000002,"15min":7.0129999999999999},"last":0.33600000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738812,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":826440,"kb_used_data":6392,"kb_used_omap":88,"kb_used_meta":819943,"kb_avail":93545400,"statfs":{"total":96636764160,"available":95790489600,"internally_reserved":0,"allocated":6545408,"data_stored":3697286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":90523,"internal_metadata":839622245},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 11:14:33 2023","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.42799999999999999,"15min":0.41099999999999998},"min":{"1min":0.151,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":1.2809999999999999,"5min":3.5739999999999998,"15min":8.2479999999999993},"last":0.223},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.48299999999999998,"15min":0.49199999999999999},"min":{"1min":0.20699999999999999,"5min":0.182,"15min":0.17999999999999999},"max":{"1min":1.323,"5min":3.6080000000000001,"15min":8.2870000000000008},"last":0.28399999999999997}]},{"osd":2,"last update":"Wed Oct 11 11:19:41 2023","interfaces":[{"interface":"back","average":{"1min":0.314,"5min":0.35399999999999998,"15min":0.35899999999999999},"min":{"1min":0.186,"5min":0.14699999999999999,"15min":0.14699999999999999},"max":{"1min":0.45500000000000002,"5min":7.4829999999999997,"15min":10.617000000000001},"last":0.28100000000000003},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.39400000000000002,"15min":0.35099999999999998},"min":{"1min":0.21199999999999999,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.49099999999999999,"5min":7.5350000000000001,"15min":7.5350000000000001},"last":0.317}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-11T11:20:20.087 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-10-11T11:20:20.087 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 517, 'reported_epoch': 500, 'state': 'active+undersized', 'last_fresh': '2023-10-11T11:19:02.772828+0000', 'last_change': '2023-10-11T11:15:31.661747+0000', 'last_active': '2023-10-11T11:19:02.772828+0000', 'last_peered': '2023-10-11T11:19:02.772828+0000', 'last_clean': '2023-10-11T11:15:27.638215+0000', 'last_became_active': '2023-10-11T11:15:31.661747+0000', 'last_became_peered': '2023-10-11T11:15:31.661747+0000', 'last_unstale': '2023-10-11T11:19:02.772828+0000', 'last_undegraded': '2023-10-11T11:19:02.772828+0000', 'last_fullsized': '2023-10-11T11:15:31.649336+0000', 'mapping_epoch': 480, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_clean_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-12T22:15:25.047039+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-10-11T11:20:20.087 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-10-11T11:20:20.088 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 29, 'reported_epoch': 500, 'state': 'active+undersized', 'last_fresh': '2023-10-11T11:19:02.774213+0000', 'last_change': '2023-10-11T11:15:31.671363+0000', 'last_active': '2023-10-11T11:19:02.774213+0000', 'last_peered': '2023-10-11T11:19:02.774213+0000', 'last_clean': '2023-10-11T10:43:40.751300+0000', 'last_became_active': '2023-10-11T11:15:31.671363+0000', 'last_became_peered': '2023-10-11T11:15:31.671363+0000', 'last_unstale': '2023-10-11T11:19:02.774213+0000', 'last_undegraded': '2023-10-11T11:19:02.774213+0000', 'last_fullsized': '2023-10-11T11:15:31.652001+0000', 'mapping_epoch': 480, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_clean_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-12T21:01:53.884964+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-10-11T11:20:20.088 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-10-11T11:20:20.088 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "11'97", 'reported_seq': 648, 'reported_epoch': 500, 'state': 'active+undersized+degraded', 'last_fresh': '2023-10-11T11:19:02.769176+0000', 'last_change': '2023-10-11T11:15:31.654022+0000', 'last_active': '2023-10-11T11:19:02.769176+0000', 'last_peered': '2023-10-11T11:19:02.769176+0000', 'last_clean': '2023-10-11T11:15:27.636884+0000', 'last_became_active': '2023-10-11T11:15:31.654022+0000', 'last_became_peered': '2023-10-11T11:15:31.654022+0000', 'last_unstale': '2023-10-11T11:19:02.769176+0000', 'last_undegraded': '2023-10-11T11:15:31.646946+0000', 'last_fullsized': '2023-10-11T11:15:31.646751+0000', 'mapping_epoch': 480, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-11T10:43:36.739637+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-11T10:43:36.739637+0000', 'last_clean_scrub_stamp': '2023-10-11T10:43:36.739637+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-12T11:33:33.696447+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-10-11T11:20:20.089 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.0 is not active+clean 2023-10-11T11:20:20.089 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 520, 'reported_epoch': 500, 'state': 'active+clean+remapped', 'last_fresh': '2023-10-11T11:19:03.429429+0000', 'last_change': '2023-10-11T11:15:33.660200+0000', 'last_active': '2023-10-11T11:19:03.429429+0000', 'last_peered': '2023-10-11T11:19:03.429429+0000', 'last_clean': '2023-10-11T11:19:03.429429+0000', 'last_became_active': '2023-10-11T11:15:33.659619+0000', 'last_became_peered': '2023-10-11T11:15:33.659619+0000', 'last_unstale': '2023-10-11T11:19:03.429429+0000', 'last_undegraded': '2023-10-11T11:19:03.429429+0000', 'last_fullsized': '2023-10-11T11:19:03.429429+0000', 'mapping_epoch': 482, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 483, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_clean_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-12T17:21:16.866392+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2, 1], 'avail_no_missing': ['2', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-10-11T11:20:20.089 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-10-11T11:20:20.089 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 517, 'reported_epoch': 500, 'state': 'active+undersized', 'last_fresh': '2023-10-11T11:19:02.775128+0000', 'last_change': '2023-10-11T11:15:31.670813+0000', 'last_active': '2023-10-11T11:19:02.775128+0000', 'last_peered': '2023-10-11T11:19:02.775128+0000', 'last_clean': '2023-10-11T11:15:27.638219+0000', 'last_became_active': '2023-10-11T11:15:31.670813+0000', 'last_became_peered': '2023-10-11T11:15:31.670813+0000', 'last_unstale': '2023-10-11T11:19:02.775128+0000', 'last_undegraded': '2023-10-11T11:19:02.775128+0000', 'last_fullsized': '2023-10-11T11:15:31.658785+0000', 'mapping_epoch': 480, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_clean_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-12T21:41:50.872346+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-10-11T11:20:20.090 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-10-11T11:20:20.090 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 516, 'reported_epoch': 499, 'state': 'active+undersized', 'last_fresh': '2023-10-11T11:19:00.631668+0000', 'last_change': '2023-10-11T11:15:31.666378+0000', 'last_active': '2023-10-11T11:19:00.631668+0000', 'last_peered': '2023-10-11T11:19:00.631668+0000', 'last_clean': '2023-10-11T11:15:27.638303+0000', 'last_became_active': '2023-10-11T11:15:31.666378+0000', 'last_became_peered': '2023-10-11T11:15:31.666378+0000', 'last_unstale': '2023-10-11T11:19:00.631668+0000', 'last_undegraded': '2023-10-11T11:19:00.631668+0000', 'last_fullsized': '2023-10-11T11:15:31.649186+0000', 'mapping_epoch': 480, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_clean_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-12T15:35:19.331209+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-10-11T11:20:20.090 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-10-11T11:20:20.091 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 516, 'reported_epoch': 499, 'state': 'active+undersized', 'last_fresh': '2023-10-11T11:19:00.636960+0000', 'last_change': '2023-10-11T11:15:31.679465+0000', 'last_active': '2023-10-11T11:19:00.636960+0000', 'last_peered': '2023-10-11T11:19:00.636960+0000', 'last_clean': '2023-10-11T11:15:27.638749+0000', 'last_became_active': '2023-10-11T11:15:31.679465+0000', 'last_became_peered': '2023-10-11T11:15:31.679465+0000', 'last_unstale': '2023-10-11T11:19:00.636960+0000', 'last_undegraded': '2023-10-11T11:19:00.636960+0000', 'last_fullsized': '2023-10-11T11:15:31.659135+0000', 'mapping_epoch': 480, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_clean_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-12T16:03:42.656692+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-10-11T11:20:20.091 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-10-11T11:20:20.485 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T11:20:20.486 INFO:teuthology.orchestra.run.smithi111.stdout:{"fsid":"022ce6f2-9a5b-45ad-8b77-a1332d92f228","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2215,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":500,"num_osds":3,"num_up_osds":2,"osd_up_since":1697022930,"num_in_osds":3,"osd_in_since":1697021005,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":2},{"state_name":"active+undersized+degraded","count":1},{"state_name":"active+clean+remapped","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3170594816,"bytes_avail":286739697664,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":36,"modified":"2023-10-11T11:19:04.644408+0000","services":{}},"progress_events":{}} 2023-10-11T11:20:20.499 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-10-11T11:20:20.500 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-10-11T11:20:20.816 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T11:20:20.819 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2023-10-11T11:20:20.830 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":1601,"stamp":"2023-10-11T11:20:20.662798+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":11,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":107,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3096284,"kb_used_data":418996,"kb_used_omap":292,"kb_used_meta":2676955,"kb_avail":280019236,"statfs":{"total":289910292480,"available":286739697664,"internally_reserved":0,"allocated":429051904,"data_stored":420453125,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299441,"internal_metadata":2741202511},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002857"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":517,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-10-11T11:19:02.772828+0000","last_change":"2023-10-11T11:15:31.661747+0000","last_active":"2023-10-11T11:19:02.772828+0000","last_peered":"2023-10-11T11:19:02.772828+0000","last_clean":"2023-10-11T11:15:27.638215+0000","last_became_active":"2023-10-11T11:15:31.661747+0000","last_became_peered":"2023-10-11T11:15:31.661747+0000","last_unstale":"2023-10-11T11:19:02.772828+0000","last_undegraded":"2023-10-11T11:19:02.772828+0000","last_fullsized":"2023-10-11T11:15:31.649336+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:15:25.047039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":29,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-10-11T11:19:02.774213+0000","last_change":"2023-10-11T11:15:31.671363+0000","last_active":"2023-10-11T11:19:02.774213+0000","last_peered":"2023-10-11T11:19:02.774213+0000","last_clean":"2023-10-11T10:43:40.751300+0000","last_became_active":"2023-10-11T11:15:31.671363+0000","last_became_peered":"2023-10-11T11:15:31.671363+0000","last_unstale":"2023-10-11T11:19:02.774213+0000","last_undegraded":"2023-10-11T11:19:02.774213+0000","last_fullsized":"2023-10-11T11:15:31.652001+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:01:53.884964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":512,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-10-11T11:19:03.429640+0000","last_change":"2023-10-11T10:43:41.772781+0000","last_active":"2023-10-11T11:19:03.429640+0000","last_peered":"2023-10-11T11:19:03.429640+0000","last_clean":"2023-10-11T11:19:03.429640+0000","last_became_active":"2023-10-11T10:43:41.772443+0000","last_became_peered":"2023-10-11T10:43:41.772443+0000","last_unstale":"2023-10-11T11:19:03.429640+0000","last_undegraded":"2023-10-11T11:19:03.429640+0000","last_fullsized":"2023-10-11T11:19:03.429640+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T14:15:30.547284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":513,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-10-11T11:19:02.772211+0000","last_change":"2023-10-11T10:43:41.772615+0000","last_active":"2023-10-11T11:19:02.772211+0000","last_peered":"2023-10-11T11:19:02.772211+0000","last_clean":"2023-10-11T11:19:02.772211+0000","last_became_active":"2023-10-11T10:43:41.772290+0000","last_became_peered":"2023-10-11T10:43:41.772290+0000","last_unstale":"2023-10-11T11:19:02.772211+0000","last_undegraded":"2023-10-11T11:19:02.772211+0000","last_fullsized":"2023-10-11T11:19:02.772211+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:26:06.909963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":648,"reported_epoch":500,"state":"active+undersized+degraded","last_fresh":"2023-10-11T11:19:02.769176+0000","last_change":"2023-10-11T11:15:31.654022+0000","last_active":"2023-10-11T11:19:02.769176+0000","last_peered":"2023-10-11T11:19:02.769176+0000","last_clean":"2023-10-11T11:15:27.636884+0000","last_became_active":"2023-10-11T11:15:31.654022+0000","last_became_peered":"2023-10-11T11:15:31.654022+0000","last_unstale":"2023-10-11T11:19:02.769176+0000","last_undegraded":"2023-10-11T11:15:31.646946+0000","last_fullsized":"2023-10-11T11:15:31.646751+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:36.739637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:36.739637+0000","last_clean_scrub_stamp":"2023-10-11T10:43:36.739637+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T11:33:33.696447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":520,"reported_epoch":500,"state":"active+clean+remapped","last_fresh":"2023-10-11T11:19:03.429429+0000","last_change":"2023-10-11T11:15:33.660200+0000","last_active":"2023-10-11T11:19:03.429429+0000","last_peered":"2023-10-11T11:19:03.429429+0000","last_clean":"2023-10-11T11:19:03.429429+0000","last_became_active":"2023-10-11T11:15:33.659619+0000","last_became_peered":"2023-10-11T11:15:33.659619+0000","last_unstale":"2023-10-11T11:19:03.429429+0000","last_undegraded":"2023-10-11T11:19:03.429429+0000","last_fullsized":"2023-10-11T11:19:03.429429+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T17:21:16.866392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":517,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-10-11T11:19:02.775128+0000","last_change":"2023-10-11T11:15:31.670813+0000","last_active":"2023-10-11T11:19:02.775128+0000","last_peered":"2023-10-11T11:19:02.775128+0000","last_clean":"2023-10-11T11:15:27.638219+0000","last_became_active":"2023-10-11T11:15:31.670813+0000","last_became_peered":"2023-10-11T11:15:31.670813+0000","last_unstale":"2023-10-11T11:19:02.775128+0000","last_undegraded":"2023-10-11T11:19:02.775128+0000","last_fullsized":"2023-10-11T11:15:31.658785+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:41:50.872346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":516,"reported_epoch":499,"state":"active+undersized","last_fresh":"2023-10-11T11:19:00.631668+0000","last_change":"2023-10-11T11:15:31.666378+0000","last_active":"2023-10-11T11:19:00.631668+0000","last_peered":"2023-10-11T11:19:00.631668+0000","last_clean":"2023-10-11T11:15:27.638303+0000","last_became_active":"2023-10-11T11:15:31.666378+0000","last_became_peered":"2023-10-11T11:15:31.666378+0000","last_unstale":"2023-10-11T11:19:00.631668+0000","last_undegraded":"2023-10-11T11:19:00.631668+0000","last_fullsized":"2023-10-11T11:15:31.649186+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T15:35:19.331209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":516,"reported_epoch":499,"state":"active+undersized","last_fresh":"2023-10-11T11:19:00.636960+0000","last_change":"2023-10-11T11:15:31.679465+0000","last_active":"2023-10-11T11:19:00.636960+0000","last_peered":"2023-10-11T11:19:00.636960+0000","last_clean":"2023-10-11T11:15:27.638749+0000","last_became_active":"2023-10-11T11:15:31.679465+0000","last_became_peered":"2023-10-11T11:15:31.679465+0000","last_unstale":"2023-10-11T11:19:00.636960+0000","last_undegraded":"2023-10-11T11:19:00.636960+0000","last_fullsized":"2023-10-11T11:15:31.659135+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T16:03:42.656692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":11,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738812,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":785348,"kb_used_data":5812,"kb_used_omap":96,"kb_used_meta":779423,"kb_avail":93586492,"statfs":{"total":96636764160,"available":95832567808,"internally_reserved":0,"allocated":5951488,"data_stored":3106918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":98609,"internal_metadata":798129871},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 11:14:37 2023","interfaces":[{"interface":"back","average":{"1min":0.314,"5min":0.316,"15min":0.443},"min":{"1min":0.24399999999999999,"5min":0.14799999999999999,"15min":0.14599999999999999},"max":{"1min":0.48499999999999999,"5min":1.712,"15min":9.6400000000000006},"last":0.31},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.35499999999999998,"15min":0.41299999999999998},"min":{"1min":0.20899999999999999,"5min":0.17199999999999999,"15min":0.16400000000000001},"max":{"1min":3.8180000000000001,"5min":3.8180000000000001,"15min":6.3200000000000003},"last":0.26500000000000001}]},{"osd":1,"last update":"Wed Oct 11 11:19:40 2023","interfaces":[{"interface":"back","average":{"1min":0.29599999999999999,"5min":0.28499999999999998,"15min":0.32300000000000001},"min":{"1min":0.17699999999999999,"5min":0.154,"15min":0.154},"max":{"1min":0.40600000000000003,"5min":0.41999999999999998,"15min":2.9540000000000002},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.317,"5min":0.31,"15min":0.36199999999999999},"min":{"1min":0.20899999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.45000000000000001,"5min":0.49099999999999999,"15min":3.2719999999999998},"last":0.36699999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738756,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1484496,"kb_used_data":406792,"kb_used_omap":107,"kb_used_meta":1077588,"kb_avail":92887344,"statfs":{"total":96636764160,"available":95116640256,"internally_reserved":0,"allocated":416555008,"data_stored":413648921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110309,"internal_metadata":1103450395},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Oct 11 11:14:30 2023","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.35399999999999998,"15min":0.40600000000000003},"min":{"1min":0.18099999999999999,"5min":0.155,"15min":0.14899999999999999},"max":{"1min":1.4319999999999999,"5min":2.3460000000000001,"15min":6.593},"last":0.26200000000000001},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.39100000000000001,"15min":0.48199999999999998},"min":{"1min":0.189,"5min":0.189,"15min":0.183},"max":{"1min":2.2120000000000002,"5min":2.4089999999999998,"15min":6.5960000000000001},"last":0.316}]},{"osd":2,"last update":"Wed Oct 11 11:14:30 2023","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.35499999999999998,"15min":0.47099999999999997},"min":{"1min":0.20300000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":1.2490000000000001,"5min":2.665,"15min":8.4730000000000008},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.375,"15min":0.442},"min":{"1min":0.23200000000000001,"5min":0.20499999999999999,"15min":0.19600000000000001},"max":{"1min":1.28,"5min":2.6230000000000002,"15min":7.0129999999999999},"last":0.33600000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738813,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":826440,"kb_used_data":6392,"kb_used_omap":88,"kb_used_meta":819943,"kb_avail":93545400,"statfs":{"total":96636764160,"available":95790489600,"internally_reserved":0,"allocated":6545408,"data_stored":3697286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":90523,"internal_metadata":839622245},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 11:14:33 2023","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.42799999999999999,"15min":0.41099999999999998},"min":{"1min":0.151,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":1.2809999999999999,"5min":3.5739999999999998,"15min":8.2479999999999993},"last":0.223},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.48299999999999998,"15min":0.49199999999999999},"min":{"1min":0.20699999999999999,"5min":0.182,"15min":0.17999999999999999},"max":{"1min":1.323,"5min":3.6080000000000001,"15min":8.2870000000000008},"last":0.28399999999999997}]},{"osd":2,"last update":"Wed Oct 11 11:19:41 2023","interfaces":[{"interface":"back","average":{"1min":0.314,"5min":0.35399999999999998,"15min":0.35899999999999999},"min":{"1min":0.186,"5min":0.14699999999999999,"15min":0.14699999999999999},"max":{"1min":0.45500000000000002,"5min":7.4829999999999997,"15min":10.617000000000001},"last":0.28499999999999998},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.39400000000000002,"15min":0.35099999999999998},"min":{"1min":0.21199999999999999,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.49099999999999999,"5min":7.5350000000000001,"15min":7.5350000000000001},"last":0.249}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-11T11:20:23.832 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-10-11T11:20:23.966 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~292s 2023-10-11T11:20:24.163 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T11:20:24.165 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2023-10-11T11:20:24.176 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":1602,"stamp":"2023-10-11T11:20:22.663353+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":11,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":107,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3096284,"kb_used_data":418996,"kb_used_omap":292,"kb_used_meta":2676955,"kb_avail":280019236,"statfs":{"total":289910292480,"available":286739697664,"internally_reserved":0,"allocated":429051904,"data_stored":420453125,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299441,"internal_metadata":2741202511},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002830"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":517,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-10-11T11:19:02.772828+0000","last_change":"2023-10-11T11:15:31.661747+0000","last_active":"2023-10-11T11:19:02.772828+0000","last_peered":"2023-10-11T11:19:02.772828+0000","last_clean":"2023-10-11T11:15:27.638215+0000","last_became_active":"2023-10-11T11:15:31.661747+0000","last_became_peered":"2023-10-11T11:15:31.661747+0000","last_unstale":"2023-10-11T11:19:02.772828+0000","last_undegraded":"2023-10-11T11:19:02.772828+0000","last_fullsized":"2023-10-11T11:15:31.649336+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:15:25.047039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":29,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-10-11T11:19:02.774213+0000","last_change":"2023-10-11T11:15:31.671363+0000","last_active":"2023-10-11T11:19:02.774213+0000","last_peered":"2023-10-11T11:19:02.774213+0000","last_clean":"2023-10-11T10:43:40.751300+0000","last_became_active":"2023-10-11T11:15:31.671363+0000","last_became_peered":"2023-10-11T11:15:31.671363+0000","last_unstale":"2023-10-11T11:19:02.774213+0000","last_undegraded":"2023-10-11T11:19:02.774213+0000","last_fullsized":"2023-10-11T11:15:31.652001+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:01:53.884964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":512,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-10-11T11:19:03.429640+0000","last_change":"2023-10-11T10:43:41.772781+0000","last_active":"2023-10-11T11:19:03.429640+0000","last_peered":"2023-10-11T11:19:03.429640+0000","last_clean":"2023-10-11T11:19:03.429640+0000","last_became_active":"2023-10-11T10:43:41.772443+0000","last_became_peered":"2023-10-11T10:43:41.772443+0000","last_unstale":"2023-10-11T11:19:03.429640+0000","last_undegraded":"2023-10-11T11:19:03.429640+0000","last_fullsized":"2023-10-11T11:19:03.429640+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T14:15:30.547284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":513,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-10-11T11:19:02.772211+0000","last_change":"2023-10-11T10:43:41.772615+0000","last_active":"2023-10-11T11:19:02.772211+0000","last_peered":"2023-10-11T11:19:02.772211+0000","last_clean":"2023-10-11T11:19:02.772211+0000","last_became_active":"2023-10-11T10:43:41.772290+0000","last_became_peered":"2023-10-11T10:43:41.772290+0000","last_unstale":"2023-10-11T11:19:02.772211+0000","last_undegraded":"2023-10-11T11:19:02.772211+0000","last_fullsized":"2023-10-11T11:19:02.772211+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:26:06.909963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":648,"reported_epoch":500,"state":"active+undersized+degraded","last_fresh":"2023-10-11T11:19:02.769176+0000","last_change":"2023-10-11T11:15:31.654022+0000","last_active":"2023-10-11T11:19:02.769176+0000","last_peered":"2023-10-11T11:19:02.769176+0000","last_clean":"2023-10-11T11:15:27.636884+0000","last_became_active":"2023-10-11T11:15:31.654022+0000","last_became_peered":"2023-10-11T11:15:31.654022+0000","last_unstale":"2023-10-11T11:19:02.769176+0000","last_undegraded":"2023-10-11T11:15:31.646946+0000","last_fullsized":"2023-10-11T11:15:31.646751+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:36.739637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:36.739637+0000","last_clean_scrub_stamp":"2023-10-11T10:43:36.739637+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T11:33:33.696447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":520,"reported_epoch":500,"state":"active+clean+remapped","last_fresh":"2023-10-11T11:19:03.429429+0000","last_change":"2023-10-11T11:15:33.660200+0000","last_active":"2023-10-11T11:19:03.429429+0000","last_peered":"2023-10-11T11:19:03.429429+0000","last_clean":"2023-10-11T11:19:03.429429+0000","last_became_active":"2023-10-11T11:15:33.659619+0000","last_became_peered":"2023-10-11T11:15:33.659619+0000","last_unstale":"2023-10-11T11:19:03.429429+0000","last_undegraded":"2023-10-11T11:19:03.429429+0000","last_fullsized":"2023-10-11T11:19:03.429429+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T17:21:16.866392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":517,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-10-11T11:19:02.775128+0000","last_change":"2023-10-11T11:15:31.670813+0000","last_active":"2023-10-11T11:19:02.775128+0000","last_peered":"2023-10-11T11:19:02.775128+0000","last_clean":"2023-10-11T11:15:27.638219+0000","last_became_active":"2023-10-11T11:15:31.670813+0000","last_became_peered":"2023-10-11T11:15:31.670813+0000","last_unstale":"2023-10-11T11:19:02.775128+0000","last_undegraded":"2023-10-11T11:19:02.775128+0000","last_fullsized":"2023-10-11T11:15:31.658785+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:41:50.872346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":516,"reported_epoch":499,"state":"active+undersized","last_fresh":"2023-10-11T11:19:00.631668+0000","last_change":"2023-10-11T11:15:31.666378+0000","last_active":"2023-10-11T11:19:00.631668+0000","last_peered":"2023-10-11T11:19:00.631668+0000","last_clean":"2023-10-11T11:15:27.638303+0000","last_became_active":"2023-10-11T11:15:31.666378+0000","last_became_peered":"2023-10-11T11:15:31.666378+0000","last_unstale":"2023-10-11T11:19:00.631668+0000","last_undegraded":"2023-10-11T11:19:00.631668+0000","last_fullsized":"2023-10-11T11:15:31.649186+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T15:35:19.331209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":516,"reported_epoch":499,"state":"active+undersized","last_fresh":"2023-10-11T11:19:00.636960+0000","last_change":"2023-10-11T11:15:31.679465+0000","last_active":"2023-10-11T11:19:00.636960+0000","last_peered":"2023-10-11T11:19:00.636960+0000","last_clean":"2023-10-11T11:15:27.638749+0000","last_became_active":"2023-10-11T11:15:31.679465+0000","last_became_peered":"2023-10-11T11:15:31.679465+0000","last_unstale":"2023-10-11T11:19:00.636960+0000","last_undegraded":"2023-10-11T11:19:00.636960+0000","last_fullsized":"2023-10-11T11:15:31.659135+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T16:03:42.656692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":11,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738812,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":785348,"kb_used_data":5812,"kb_used_omap":96,"kb_used_meta":779423,"kb_avail":93586492,"statfs":{"total":96636764160,"available":95832567808,"internally_reserved":0,"allocated":5951488,"data_stored":3106918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":98609,"internal_metadata":798129871},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 11:14:37 2023","interfaces":[{"interface":"back","average":{"1min":0.314,"5min":0.316,"15min":0.443},"min":{"1min":0.24399999999999999,"5min":0.14799999999999999,"15min":0.14599999999999999},"max":{"1min":0.48499999999999999,"5min":1.712,"15min":9.6400000000000006},"last":0.31},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.35499999999999998,"15min":0.41299999999999998},"min":{"1min":0.20899999999999999,"5min":0.17199999999999999,"15min":0.16400000000000001},"max":{"1min":3.8180000000000001,"5min":3.8180000000000001,"15min":6.3200000000000003},"last":0.26500000000000001}]},{"osd":1,"last update":"Wed Oct 11 11:19:40 2023","interfaces":[{"interface":"back","average":{"1min":0.29599999999999999,"5min":0.28499999999999998,"15min":0.32300000000000001},"min":{"1min":0.17699999999999999,"5min":0.154,"15min":0.154},"max":{"1min":0.40600000000000003,"5min":0.41999999999999998,"15min":2.9540000000000002},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.317,"5min":0.31,"15min":0.36199999999999999},"min":{"1min":0.20899999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.45000000000000001,"5min":0.49099999999999999,"15min":3.2719999999999998},"last":0.36699999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738756,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1484496,"kb_used_data":406792,"kb_used_omap":107,"kb_used_meta":1077588,"kb_avail":92887344,"statfs":{"total":96636764160,"available":95116640256,"internally_reserved":0,"allocated":416555008,"data_stored":413648921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110309,"internal_metadata":1103450395},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Oct 11 11:14:30 2023","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.35399999999999998,"15min":0.40600000000000003},"min":{"1min":0.18099999999999999,"5min":0.155,"15min":0.14899999999999999},"max":{"1min":1.4319999999999999,"5min":2.3460000000000001,"15min":6.593},"last":0.26200000000000001},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.39100000000000001,"15min":0.48199999999999998},"min":{"1min":0.189,"5min":0.189,"15min":0.183},"max":{"1min":2.2120000000000002,"5min":2.4089999999999998,"15min":6.5960000000000001},"last":0.316}]},{"osd":2,"last update":"Wed Oct 11 11:14:30 2023","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.35499999999999998,"15min":0.47099999999999997},"min":{"1min":0.20300000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":1.2490000000000001,"5min":2.665,"15min":8.4730000000000008},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.375,"15min":0.442},"min":{"1min":0.23200000000000001,"5min":0.20499999999999999,"15min":0.19600000000000001},"max":{"1min":1.28,"5min":2.6230000000000002,"15min":7.0129999999999999},"last":0.33600000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738813,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":826440,"kb_used_data":6392,"kb_used_omap":88,"kb_used_meta":819943,"kb_avail":93545400,"statfs":{"total":96636764160,"available":95790489600,"internally_reserved":0,"allocated":6545408,"data_stored":3697286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":90523,"internal_metadata":839622245},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 11:14:33 2023","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.42799999999999999,"15min":0.41099999999999998},"min":{"1min":0.151,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":1.2809999999999999,"5min":3.5739999999999998,"15min":8.2479999999999993},"last":0.223},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.48299999999999998,"15min":0.49199999999999999},"min":{"1min":0.20699999999999999,"5min":0.182,"15min":0.17999999999999999},"max":{"1min":1.323,"5min":3.6080000000000001,"15min":8.2870000000000008},"last":0.28399999999999997}]},{"osd":2,"last update":"Wed Oct 11 11:19:41 2023","interfaces":[{"interface":"back","average":{"1min":0.314,"5min":0.35399999999999998,"15min":0.35899999999999999},"min":{"1min":0.186,"5min":0.14699999999999999,"15min":0.14699999999999999},"max":{"1min":0.45500000000000002,"5min":7.4829999999999997,"15min":10.617000000000001},"last":0.28499999999999998},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.39400000000000002,"15min":0.35099999999999998},"min":{"1min":0.21199999999999999,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.49099999999999999,"5min":7.5350000000000001,"15min":7.5350000000000001},"last":0.249}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-11T11:20:24.178 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-10-11T11:20:24.494 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T11:20:24.496 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2023-10-11T11:20:24.507 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":1602,"stamp":"2023-10-11T11:20:22.663353+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":11,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":107,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3096284,"kb_used_data":418996,"kb_used_omap":292,"kb_used_meta":2676955,"kb_avail":280019236,"statfs":{"total":289910292480,"available":286739697664,"internally_reserved":0,"allocated":429051904,"data_stored":420453125,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299441,"internal_metadata":2741202511},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002830"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":517,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-10-11T11:19:02.772828+0000","last_change":"2023-10-11T11:15:31.661747+0000","last_active":"2023-10-11T11:19:02.772828+0000","last_peered":"2023-10-11T11:19:02.772828+0000","last_clean":"2023-10-11T11:15:27.638215+0000","last_became_active":"2023-10-11T11:15:31.661747+0000","last_became_peered":"2023-10-11T11:15:31.661747+0000","last_unstale":"2023-10-11T11:19:02.772828+0000","last_undegraded":"2023-10-11T11:19:02.772828+0000","last_fullsized":"2023-10-11T11:15:31.649336+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:15:25.047039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":29,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-10-11T11:19:02.774213+0000","last_change":"2023-10-11T11:15:31.671363+0000","last_active":"2023-10-11T11:19:02.774213+0000","last_peered":"2023-10-11T11:19:02.774213+0000","last_clean":"2023-10-11T10:43:40.751300+0000","last_became_active":"2023-10-11T11:15:31.671363+0000","last_became_peered":"2023-10-11T11:15:31.671363+0000","last_unstale":"2023-10-11T11:19:02.774213+0000","last_undegraded":"2023-10-11T11:19:02.774213+0000","last_fullsized":"2023-10-11T11:15:31.652001+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:01:53.884964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":512,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-10-11T11:19:03.429640+0000","last_change":"2023-10-11T10:43:41.772781+0000","last_active":"2023-10-11T11:19:03.429640+0000","last_peered":"2023-10-11T11:19:03.429640+0000","last_clean":"2023-10-11T11:19:03.429640+0000","last_became_active":"2023-10-11T10:43:41.772443+0000","last_became_peered":"2023-10-11T10:43:41.772443+0000","last_unstale":"2023-10-11T11:19:03.429640+0000","last_undegraded":"2023-10-11T11:19:03.429640+0000","last_fullsized":"2023-10-11T11:19:03.429640+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T14:15:30.547284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":513,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-10-11T11:19:02.772211+0000","last_change":"2023-10-11T10:43:41.772615+0000","last_active":"2023-10-11T11:19:02.772211+0000","last_peered":"2023-10-11T11:19:02.772211+0000","last_clean":"2023-10-11T11:19:02.772211+0000","last_became_active":"2023-10-11T10:43:41.772290+0000","last_became_peered":"2023-10-11T10:43:41.772290+0000","last_unstale":"2023-10-11T11:19:02.772211+0000","last_undegraded":"2023-10-11T11:19:02.772211+0000","last_fullsized":"2023-10-11T11:19:02.772211+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:26:06.909963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":648,"reported_epoch":500,"state":"active+undersized+degraded","last_fresh":"2023-10-11T11:19:02.769176+0000","last_change":"2023-10-11T11:15:31.654022+0000","last_active":"2023-10-11T11:19:02.769176+0000","last_peered":"2023-10-11T11:19:02.769176+0000","last_clean":"2023-10-11T11:15:27.636884+0000","last_became_active":"2023-10-11T11:15:31.654022+0000","last_became_peered":"2023-10-11T11:15:31.654022+0000","last_unstale":"2023-10-11T11:19:02.769176+0000","last_undegraded":"2023-10-11T11:15:31.646946+0000","last_fullsized":"2023-10-11T11:15:31.646751+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:36.739637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:36.739637+0000","last_clean_scrub_stamp":"2023-10-11T10:43:36.739637+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T11:33:33.696447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":520,"reported_epoch":500,"state":"active+clean+remapped","last_fresh":"2023-10-11T11:19:03.429429+0000","last_change":"2023-10-11T11:15:33.660200+0000","last_active":"2023-10-11T11:19:03.429429+0000","last_peered":"2023-10-11T11:19:03.429429+0000","last_clean":"2023-10-11T11:19:03.429429+0000","last_became_active":"2023-10-11T11:15:33.659619+0000","last_became_peered":"2023-10-11T11:15:33.659619+0000","last_unstale":"2023-10-11T11:19:03.429429+0000","last_undegraded":"2023-10-11T11:19:03.429429+0000","last_fullsized":"2023-10-11T11:19:03.429429+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T17:21:16.866392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":517,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-10-11T11:19:02.775128+0000","last_change":"2023-10-11T11:15:31.670813+0000","last_active":"2023-10-11T11:19:02.775128+0000","last_peered":"2023-10-11T11:19:02.775128+0000","last_clean":"2023-10-11T11:15:27.638219+0000","last_became_active":"2023-10-11T11:15:31.670813+0000","last_became_peered":"2023-10-11T11:15:31.670813+0000","last_unstale":"2023-10-11T11:19:02.775128+0000","last_undegraded":"2023-10-11T11:19:02.775128+0000","last_fullsized":"2023-10-11T11:15:31.658785+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:41:50.872346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":516,"reported_epoch":499,"state":"active+undersized","last_fresh":"2023-10-11T11:19:00.631668+0000","last_change":"2023-10-11T11:15:31.666378+0000","last_active":"2023-10-11T11:19:00.631668+0000","last_peered":"2023-10-11T11:19:00.631668+0000","last_clean":"2023-10-11T11:15:27.638303+0000","last_became_active":"2023-10-11T11:15:31.666378+0000","last_became_peered":"2023-10-11T11:15:31.666378+0000","last_unstale":"2023-10-11T11:19:00.631668+0000","last_undegraded":"2023-10-11T11:19:00.631668+0000","last_fullsized":"2023-10-11T11:15:31.649186+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T15:35:19.331209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":516,"reported_epoch":499,"state":"active+undersized","last_fresh":"2023-10-11T11:19:00.636960+0000","last_change":"2023-10-11T11:15:31.679465+0000","last_active":"2023-10-11T11:19:00.636960+0000","last_peered":"2023-10-11T11:19:00.636960+0000","last_clean":"2023-10-11T11:15:27.638749+0000","last_became_active":"2023-10-11T11:15:31.679465+0000","last_became_peered":"2023-10-11T11:15:31.679465+0000","last_unstale":"2023-10-11T11:19:00.636960+0000","last_undegraded":"2023-10-11T11:19:00.636960+0000","last_fullsized":"2023-10-11T11:15:31.659135+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T16:03:42.656692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":11,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738812,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":785348,"kb_used_data":5812,"kb_used_omap":96,"kb_used_meta":779423,"kb_avail":93586492,"statfs":{"total":96636764160,"available":95832567808,"internally_reserved":0,"allocated":5951488,"data_stored":3106918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":98609,"internal_metadata":798129871},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 11:14:37 2023","interfaces":[{"interface":"back","average":{"1min":0.314,"5min":0.316,"15min":0.443},"min":{"1min":0.24399999999999999,"5min":0.14799999999999999,"15min":0.14599999999999999},"max":{"1min":0.48499999999999999,"5min":1.712,"15min":9.6400000000000006},"last":0.31},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.35499999999999998,"15min":0.41299999999999998},"min":{"1min":0.20899999999999999,"5min":0.17199999999999999,"15min":0.16400000000000001},"max":{"1min":3.8180000000000001,"5min":3.8180000000000001,"15min":6.3200000000000003},"last":0.26500000000000001}]},{"osd":1,"last update":"Wed Oct 11 11:19:40 2023","interfaces":[{"interface":"back","average":{"1min":0.29599999999999999,"5min":0.28499999999999998,"15min":0.32300000000000001},"min":{"1min":0.17699999999999999,"5min":0.154,"15min":0.154},"max":{"1min":0.40600000000000003,"5min":0.41999999999999998,"15min":2.9540000000000002},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.317,"5min":0.31,"15min":0.36199999999999999},"min":{"1min":0.20899999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.45000000000000001,"5min":0.49099999999999999,"15min":3.2719999999999998},"last":0.36699999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738756,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1484496,"kb_used_data":406792,"kb_used_omap":107,"kb_used_meta":1077588,"kb_avail":92887344,"statfs":{"total":96636764160,"available":95116640256,"internally_reserved":0,"allocated":416555008,"data_stored":413648921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110309,"internal_metadata":1103450395},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Oct 11 11:14:30 2023","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.35399999999999998,"15min":0.40600000000000003},"min":{"1min":0.18099999999999999,"5min":0.155,"15min":0.14899999999999999},"max":{"1min":1.4319999999999999,"5min":2.3460000000000001,"15min":6.593},"last":0.26200000000000001},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.39100000000000001,"15min":0.48199999999999998},"min":{"1min":0.189,"5min":0.189,"15min":0.183},"max":{"1min":2.2120000000000002,"5min":2.4089999999999998,"15min":6.5960000000000001},"last":0.316}]},{"osd":2,"last update":"Wed Oct 11 11:14:30 2023","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.35499999999999998,"15min":0.47099999999999997},"min":{"1min":0.20300000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":1.2490000000000001,"5min":2.665,"15min":8.4730000000000008},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.375,"15min":0.442},"min":{"1min":0.23200000000000001,"5min":0.20499999999999999,"15min":0.19600000000000001},"max":{"1min":1.28,"5min":2.6230000000000002,"15min":7.0129999999999999},"last":0.33600000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738813,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":826440,"kb_used_data":6392,"kb_used_omap":88,"kb_used_meta":819943,"kb_avail":93545400,"statfs":{"total":96636764160,"available":95790489600,"internally_reserved":0,"allocated":6545408,"data_stored":3697286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":90523,"internal_metadata":839622245},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 11:14:33 2023","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.42799999999999999,"15min":0.41099999999999998},"min":{"1min":0.151,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":1.2809999999999999,"5min":3.5739999999999998,"15min":8.2479999999999993},"last":0.223},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.48299999999999998,"15min":0.49199999999999999},"min":{"1min":0.20699999999999999,"5min":0.182,"15min":0.17999999999999999},"max":{"1min":1.323,"5min":3.6080000000000001,"15min":8.2870000000000008},"last":0.28399999999999997}]},{"osd":2,"last update":"Wed Oct 11 11:19:41 2023","interfaces":[{"interface":"back","average":{"1min":0.314,"5min":0.35399999999999998,"15min":0.35899999999999999},"min":{"1min":0.186,"5min":0.14699999999999999,"15min":0.14699999999999999},"max":{"1min":0.45500000000000002,"5min":7.4829999999999997,"15min":10.617000000000001},"last":0.28499999999999998},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.39400000000000002,"15min":0.35099999999999998},"min":{"1min":0.21199999999999999,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.49099999999999999,"5min":7.5350000000000001,"15min":7.5350000000000001},"last":0.249}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-11T11:20:24.508 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-10-11T11:20:24.509 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 517, 'reported_epoch': 500, 'state': 'active+undersized', 'last_fresh': '2023-10-11T11:19:02.772828+0000', 'last_change': '2023-10-11T11:15:31.661747+0000', 'last_active': '2023-10-11T11:19:02.772828+0000', 'last_peered': '2023-10-11T11:19:02.772828+0000', 'last_clean': '2023-10-11T11:15:27.638215+0000', 'last_became_active': '2023-10-11T11:15:31.661747+0000', 'last_became_peered': '2023-10-11T11:15:31.661747+0000', 'last_unstale': '2023-10-11T11:19:02.772828+0000', 'last_undegraded': '2023-10-11T11:19:02.772828+0000', 'last_fullsized': '2023-10-11T11:15:31.649336+0000', 'mapping_epoch': 480, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_clean_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-12T22:15:25.047039+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-10-11T11:20:24.509 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-10-11T11:20:24.510 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 29, 'reported_epoch': 500, 'state': 'active+undersized', 'last_fresh': '2023-10-11T11:19:02.774213+0000', 'last_change': '2023-10-11T11:15:31.671363+0000', 'last_active': '2023-10-11T11:19:02.774213+0000', 'last_peered': '2023-10-11T11:19:02.774213+0000', 'last_clean': '2023-10-11T10:43:40.751300+0000', 'last_became_active': '2023-10-11T11:15:31.671363+0000', 'last_became_peered': '2023-10-11T11:15:31.671363+0000', 'last_unstale': '2023-10-11T11:19:02.774213+0000', 'last_undegraded': '2023-10-11T11:19:02.774213+0000', 'last_fullsized': '2023-10-11T11:15:31.652001+0000', 'mapping_epoch': 480, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_clean_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-12T21:01:53.884964+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-10-11T11:20:24.510 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-10-11T11:20:24.510 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "11'97", 'reported_seq': 648, 'reported_epoch': 500, 'state': 'active+undersized+degraded', 'last_fresh': '2023-10-11T11:19:02.769176+0000', 'last_change': '2023-10-11T11:15:31.654022+0000', 'last_active': '2023-10-11T11:19:02.769176+0000', 'last_peered': '2023-10-11T11:19:02.769176+0000', 'last_clean': '2023-10-11T11:15:27.636884+0000', 'last_became_active': '2023-10-11T11:15:31.654022+0000', 'last_became_peered': '2023-10-11T11:15:31.654022+0000', 'last_unstale': '2023-10-11T11:19:02.769176+0000', 'last_undegraded': '2023-10-11T11:15:31.646946+0000', 'last_fullsized': '2023-10-11T11:15:31.646751+0000', 'mapping_epoch': 480, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-11T10:43:36.739637+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-11T10:43:36.739637+0000', 'last_clean_scrub_stamp': '2023-10-11T10:43:36.739637+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-12T11:33:33.696447+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-10-11T11:20:24.511 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.0 is not active+clean 2023-10-11T11:20:24.511 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 520, 'reported_epoch': 500, 'state': 'active+clean+remapped', 'last_fresh': '2023-10-11T11:19:03.429429+0000', 'last_change': '2023-10-11T11:15:33.660200+0000', 'last_active': '2023-10-11T11:19:03.429429+0000', 'last_peered': '2023-10-11T11:19:03.429429+0000', 'last_clean': '2023-10-11T11:19:03.429429+0000', 'last_became_active': '2023-10-11T11:15:33.659619+0000', 'last_became_peered': '2023-10-11T11:15:33.659619+0000', 'last_unstale': '2023-10-11T11:19:03.429429+0000', 'last_undegraded': '2023-10-11T11:19:03.429429+0000', 'last_fullsized': '2023-10-11T11:19:03.429429+0000', 'mapping_epoch': 482, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 483, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_clean_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-12T17:21:16.866392+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2, 1], 'avail_no_missing': ['2', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-10-11T11:20:24.511 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-10-11T11:20:24.512 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 517, 'reported_epoch': 500, 'state': 'active+undersized', 'last_fresh': '2023-10-11T11:19:02.775128+0000', 'last_change': '2023-10-11T11:15:31.670813+0000', 'last_active': '2023-10-11T11:19:02.775128+0000', 'last_peered': '2023-10-11T11:19:02.775128+0000', 'last_clean': '2023-10-11T11:15:27.638219+0000', 'last_became_active': '2023-10-11T11:15:31.670813+0000', 'last_became_peered': '2023-10-11T11:15:31.670813+0000', 'last_unstale': '2023-10-11T11:19:02.775128+0000', 'last_undegraded': '2023-10-11T11:19:02.775128+0000', 'last_fullsized': '2023-10-11T11:15:31.658785+0000', 'mapping_epoch': 480, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_clean_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-12T21:41:50.872346+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-10-11T11:20:24.512 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-10-11T11:20:24.512 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 516, 'reported_epoch': 499, 'state': 'active+undersized', 'last_fresh': '2023-10-11T11:19:00.631668+0000', 'last_change': '2023-10-11T11:15:31.666378+0000', 'last_active': '2023-10-11T11:19:00.631668+0000', 'last_peered': '2023-10-11T11:19:00.631668+0000', 'last_clean': '2023-10-11T11:15:27.638303+0000', 'last_became_active': '2023-10-11T11:15:31.666378+0000', 'last_became_peered': '2023-10-11T11:15:31.666378+0000', 'last_unstale': '2023-10-11T11:19:00.631668+0000', 'last_undegraded': '2023-10-11T11:19:00.631668+0000', 'last_fullsized': '2023-10-11T11:15:31.649186+0000', 'mapping_epoch': 480, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_clean_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-12T15:35:19.331209+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-10-11T11:20:24.513 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-10-11T11:20:24.513 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 516, 'reported_epoch': 499, 'state': 'active+undersized', 'last_fresh': '2023-10-11T11:19:00.636960+0000', 'last_change': '2023-10-11T11:15:31.679465+0000', 'last_active': '2023-10-11T11:19:00.636960+0000', 'last_peered': '2023-10-11T11:19:00.636960+0000', 'last_clean': '2023-10-11T11:15:27.638749+0000', 'last_became_active': '2023-10-11T11:15:31.679465+0000', 'last_became_peered': '2023-10-11T11:15:31.679465+0000', 'last_unstale': '2023-10-11T11:19:00.636960+0000', 'last_undegraded': '2023-10-11T11:19:00.636960+0000', 'last_fullsized': '2023-10-11T11:15:31.659135+0000', 'mapping_epoch': 480, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_clean_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-12T16:03:42.656692+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-10-11T11:20:24.514 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-10-11T11:20:24.913 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T11:20:24.914 INFO:teuthology.orchestra.run.smithi111.stdout:{"fsid":"022ce6f2-9a5b-45ad-8b77-a1332d92f228","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2220,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":500,"num_osds":3,"num_up_osds":2,"osd_up_since":1697022930,"num_in_osds":3,"osd_in_since":1697021005,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":2},{"state_name":"active+undersized+degraded","count":1},{"state_name":"active+clean+remapped","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3170594816,"bytes_avail":286739697664,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":36,"modified":"2023-10-11T11:19:04.644408+0000","services":{}},"progress_events":{}} 2023-10-11T11:20:24.928 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-10-11T11:20:24.928 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-10-11T11:20:25.247 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T11:20:25.249 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2023-10-11T11:20:25.261 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":1603,"stamp":"2023-10-11T11:20:24.663907+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":11,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":107,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3096284,"kb_used_data":418996,"kb_used_omap":292,"kb_used_meta":2676955,"kb_avail":280019236,"statfs":{"total":289910292480,"available":286739697664,"internally_reserved":0,"allocated":429051904,"data_stored":420453125,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299441,"internal_metadata":2741202511},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002934"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":517,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-10-11T11:19:02.772828+0000","last_change":"2023-10-11T11:15:31.661747+0000","last_active":"2023-10-11T11:19:02.772828+0000","last_peered":"2023-10-11T11:19:02.772828+0000","last_clean":"2023-10-11T11:15:27.638215+0000","last_became_active":"2023-10-11T11:15:31.661747+0000","last_became_peered":"2023-10-11T11:15:31.661747+0000","last_unstale":"2023-10-11T11:19:02.772828+0000","last_undegraded":"2023-10-11T11:19:02.772828+0000","last_fullsized":"2023-10-11T11:15:31.649336+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:15:25.047039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":29,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-10-11T11:19:02.774213+0000","last_change":"2023-10-11T11:15:31.671363+0000","last_active":"2023-10-11T11:19:02.774213+0000","last_peered":"2023-10-11T11:19:02.774213+0000","last_clean":"2023-10-11T10:43:40.751300+0000","last_became_active":"2023-10-11T11:15:31.671363+0000","last_became_peered":"2023-10-11T11:15:31.671363+0000","last_unstale":"2023-10-11T11:19:02.774213+0000","last_undegraded":"2023-10-11T11:19:02.774213+0000","last_fullsized":"2023-10-11T11:15:31.652001+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:01:53.884964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":512,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-10-11T11:19:03.429640+0000","last_change":"2023-10-11T10:43:41.772781+0000","last_active":"2023-10-11T11:19:03.429640+0000","last_peered":"2023-10-11T11:19:03.429640+0000","last_clean":"2023-10-11T11:19:03.429640+0000","last_became_active":"2023-10-11T10:43:41.772443+0000","last_became_peered":"2023-10-11T10:43:41.772443+0000","last_unstale":"2023-10-11T11:19:03.429640+0000","last_undegraded":"2023-10-11T11:19:03.429640+0000","last_fullsized":"2023-10-11T11:19:03.429640+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T14:15:30.547284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":513,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-10-11T11:19:02.772211+0000","last_change":"2023-10-11T10:43:41.772615+0000","last_active":"2023-10-11T11:19:02.772211+0000","last_peered":"2023-10-11T11:19:02.772211+0000","last_clean":"2023-10-11T11:19:02.772211+0000","last_became_active":"2023-10-11T10:43:41.772290+0000","last_became_peered":"2023-10-11T10:43:41.772290+0000","last_unstale":"2023-10-11T11:19:02.772211+0000","last_undegraded":"2023-10-11T11:19:02.772211+0000","last_fullsized":"2023-10-11T11:19:02.772211+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:26:06.909963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":648,"reported_epoch":500,"state":"active+undersized+degraded","last_fresh":"2023-10-11T11:19:02.769176+0000","last_change":"2023-10-11T11:15:31.654022+0000","last_active":"2023-10-11T11:19:02.769176+0000","last_peered":"2023-10-11T11:19:02.769176+0000","last_clean":"2023-10-11T11:15:27.636884+0000","last_became_active":"2023-10-11T11:15:31.654022+0000","last_became_peered":"2023-10-11T11:15:31.654022+0000","last_unstale":"2023-10-11T11:19:02.769176+0000","last_undegraded":"2023-10-11T11:15:31.646946+0000","last_fullsized":"2023-10-11T11:15:31.646751+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:36.739637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:36.739637+0000","last_clean_scrub_stamp":"2023-10-11T10:43:36.739637+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T11:33:33.696447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":520,"reported_epoch":500,"state":"active+clean+remapped","last_fresh":"2023-10-11T11:19:03.429429+0000","last_change":"2023-10-11T11:15:33.660200+0000","last_active":"2023-10-11T11:19:03.429429+0000","last_peered":"2023-10-11T11:19:03.429429+0000","last_clean":"2023-10-11T11:19:03.429429+0000","last_became_active":"2023-10-11T11:15:33.659619+0000","last_became_peered":"2023-10-11T11:15:33.659619+0000","last_unstale":"2023-10-11T11:19:03.429429+0000","last_undegraded":"2023-10-11T11:19:03.429429+0000","last_fullsized":"2023-10-11T11:19:03.429429+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T17:21:16.866392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":517,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-10-11T11:19:02.775128+0000","last_change":"2023-10-11T11:15:31.670813+0000","last_active":"2023-10-11T11:19:02.775128+0000","last_peered":"2023-10-11T11:19:02.775128+0000","last_clean":"2023-10-11T11:15:27.638219+0000","last_became_active":"2023-10-11T11:15:31.670813+0000","last_became_peered":"2023-10-11T11:15:31.670813+0000","last_unstale":"2023-10-11T11:19:02.775128+0000","last_undegraded":"2023-10-11T11:19:02.775128+0000","last_fullsized":"2023-10-11T11:15:31.658785+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:41:50.872346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":516,"reported_epoch":499,"state":"active+undersized","last_fresh":"2023-10-11T11:19:00.631668+0000","last_change":"2023-10-11T11:15:31.666378+0000","last_active":"2023-10-11T11:19:00.631668+0000","last_peered":"2023-10-11T11:19:00.631668+0000","last_clean":"2023-10-11T11:15:27.638303+0000","last_became_active":"2023-10-11T11:15:31.666378+0000","last_became_peered":"2023-10-11T11:15:31.666378+0000","last_unstale":"2023-10-11T11:19:00.631668+0000","last_undegraded":"2023-10-11T11:19:00.631668+0000","last_fullsized":"2023-10-11T11:15:31.649186+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T15:35:19.331209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":516,"reported_epoch":499,"state":"active+undersized","last_fresh":"2023-10-11T11:19:00.636960+0000","last_change":"2023-10-11T11:15:31.679465+0000","last_active":"2023-10-11T11:19:00.636960+0000","last_peered":"2023-10-11T11:19:00.636960+0000","last_clean":"2023-10-11T11:15:27.638749+0000","last_became_active":"2023-10-11T11:15:31.679465+0000","last_became_peered":"2023-10-11T11:15:31.679465+0000","last_unstale":"2023-10-11T11:19:00.636960+0000","last_undegraded":"2023-10-11T11:19:00.636960+0000","last_fullsized":"2023-10-11T11:15:31.659135+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T16:03:42.656692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":11,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738812,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":785348,"kb_used_data":5812,"kb_used_omap":96,"kb_used_meta":779423,"kb_avail":93586492,"statfs":{"total":96636764160,"available":95832567808,"internally_reserved":0,"allocated":5951488,"data_stored":3106918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":98609,"internal_metadata":798129871},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 11:14:37 2023","interfaces":[{"interface":"back","average":{"1min":0.314,"5min":0.316,"15min":0.443},"min":{"1min":0.24399999999999999,"5min":0.14799999999999999,"15min":0.14599999999999999},"max":{"1min":0.48499999999999999,"5min":1.712,"15min":9.6400000000000006},"last":0.31},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.35499999999999998,"15min":0.41299999999999998},"min":{"1min":0.20899999999999999,"5min":0.17199999999999999,"15min":0.16400000000000001},"max":{"1min":3.8180000000000001,"5min":3.8180000000000001,"15min":6.3200000000000003},"last":0.26500000000000001}]},{"osd":1,"last update":"Wed Oct 11 11:19:40 2023","interfaces":[{"interface":"back","average":{"1min":0.29599999999999999,"5min":0.28499999999999998,"15min":0.32300000000000001},"min":{"1min":0.17699999999999999,"5min":0.154,"15min":0.154},"max":{"1min":0.40600000000000003,"5min":0.41999999999999998,"15min":2.9540000000000002},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.317,"5min":0.31,"15min":0.36199999999999999},"min":{"1min":0.20899999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.45000000000000001,"5min":0.49099999999999999,"15min":3.2719999999999998},"last":0.36699999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738756,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1484496,"kb_used_data":406792,"kb_used_omap":107,"kb_used_meta":1077588,"kb_avail":92887344,"statfs":{"total":96636764160,"available":95116640256,"internally_reserved":0,"allocated":416555008,"data_stored":413648921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110309,"internal_metadata":1103450395},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Oct 11 11:14:30 2023","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.35399999999999998,"15min":0.40600000000000003},"min":{"1min":0.18099999999999999,"5min":0.155,"15min":0.14899999999999999},"max":{"1min":1.4319999999999999,"5min":2.3460000000000001,"15min":6.593},"last":0.26200000000000001},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.39100000000000001,"15min":0.48199999999999998},"min":{"1min":0.189,"5min":0.189,"15min":0.183},"max":{"1min":2.2120000000000002,"5min":2.4089999999999998,"15min":6.5960000000000001},"last":0.316}]},{"osd":2,"last update":"Wed Oct 11 11:14:30 2023","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.35499999999999998,"15min":0.47099999999999997},"min":{"1min":0.20300000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":1.2490000000000001,"5min":2.665,"15min":8.4730000000000008},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.375,"15min":0.442},"min":{"1min":0.23200000000000001,"5min":0.20499999999999999,"15min":0.19600000000000001},"max":{"1min":1.28,"5min":2.6230000000000002,"15min":7.0129999999999999},"last":0.33600000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738813,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":826440,"kb_used_data":6392,"kb_used_omap":88,"kb_used_meta":819943,"kb_avail":93545400,"statfs":{"total":96636764160,"available":95790489600,"internally_reserved":0,"allocated":6545408,"data_stored":3697286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":90523,"internal_metadata":839622245},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 11:14:33 2023","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.42799999999999999,"15min":0.41099999999999998},"min":{"1min":0.151,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":1.2809999999999999,"5min":3.5739999999999998,"15min":8.2479999999999993},"last":0.223},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.48299999999999998,"15min":0.49199999999999999},"min":{"1min":0.20699999999999999,"5min":0.182,"15min":0.17999999999999999},"max":{"1min":1.323,"5min":3.6080000000000001,"15min":8.2870000000000008},"last":0.28399999999999997}]},{"osd":2,"last update":"Wed Oct 11 11:19:41 2023","interfaces":[{"interface":"back","average":{"1min":0.314,"5min":0.35399999999999998,"15min":0.35899999999999999},"min":{"1min":0.186,"5min":0.14699999999999999,"15min":0.14699999999999999},"max":{"1min":0.45500000000000002,"5min":7.4829999999999997,"15min":10.617000000000001},"last":0.28499999999999998},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.39400000000000002,"15min":0.35099999999999998},"min":{"1min":0.21199999999999999,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.49099999999999999,"5min":7.5350000000000001,"15min":7.5350000000000001},"last":0.249}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-11T11:20:28.263 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-10-11T11:20:28.591 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T11:20:28.594 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2023-10-11T11:20:28.605 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":1604,"stamp":"2023-10-11T11:20:26.664383+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":11,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":107,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3096284,"kb_used_data":418996,"kb_used_omap":292,"kb_used_meta":2676955,"kb_avail":280019236,"statfs":{"total":289910292480,"available":286739697664,"internally_reserved":0,"allocated":429051904,"data_stored":420453125,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299441,"internal_metadata":2741202511},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002952"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":517,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-10-11T11:19:02.772828+0000","last_change":"2023-10-11T11:15:31.661747+0000","last_active":"2023-10-11T11:19:02.772828+0000","last_peered":"2023-10-11T11:19:02.772828+0000","last_clean":"2023-10-11T11:15:27.638215+0000","last_became_active":"2023-10-11T11:15:31.661747+0000","last_became_peered":"2023-10-11T11:15:31.661747+0000","last_unstale":"2023-10-11T11:19:02.772828+0000","last_undegraded":"2023-10-11T11:19:02.772828+0000","last_fullsized":"2023-10-11T11:15:31.649336+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:15:25.047039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":29,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-10-11T11:19:02.774213+0000","last_change":"2023-10-11T11:15:31.671363+0000","last_active":"2023-10-11T11:19:02.774213+0000","last_peered":"2023-10-11T11:19:02.774213+0000","last_clean":"2023-10-11T10:43:40.751300+0000","last_became_active":"2023-10-11T11:15:31.671363+0000","last_became_peered":"2023-10-11T11:15:31.671363+0000","last_unstale":"2023-10-11T11:19:02.774213+0000","last_undegraded":"2023-10-11T11:19:02.774213+0000","last_fullsized":"2023-10-11T11:15:31.652001+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:01:53.884964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":512,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-10-11T11:19:03.429640+0000","last_change":"2023-10-11T10:43:41.772781+0000","last_active":"2023-10-11T11:19:03.429640+0000","last_peered":"2023-10-11T11:19:03.429640+0000","last_clean":"2023-10-11T11:19:03.429640+0000","last_became_active":"2023-10-11T10:43:41.772443+0000","last_became_peered":"2023-10-11T10:43:41.772443+0000","last_unstale":"2023-10-11T11:19:03.429640+0000","last_undegraded":"2023-10-11T11:19:03.429640+0000","last_fullsized":"2023-10-11T11:19:03.429640+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T14:15:30.547284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":513,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-10-11T11:19:02.772211+0000","last_change":"2023-10-11T10:43:41.772615+0000","last_active":"2023-10-11T11:19:02.772211+0000","last_peered":"2023-10-11T11:19:02.772211+0000","last_clean":"2023-10-11T11:19:02.772211+0000","last_became_active":"2023-10-11T10:43:41.772290+0000","last_became_peered":"2023-10-11T10:43:41.772290+0000","last_unstale":"2023-10-11T11:19:02.772211+0000","last_undegraded":"2023-10-11T11:19:02.772211+0000","last_fullsized":"2023-10-11T11:19:02.772211+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:26:06.909963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":648,"reported_epoch":500,"state":"active+undersized+degraded","last_fresh":"2023-10-11T11:19:02.769176+0000","last_change":"2023-10-11T11:15:31.654022+0000","last_active":"2023-10-11T11:19:02.769176+0000","last_peered":"2023-10-11T11:19:02.769176+0000","last_clean":"2023-10-11T11:15:27.636884+0000","last_became_active":"2023-10-11T11:15:31.654022+0000","last_became_peered":"2023-10-11T11:15:31.654022+0000","last_unstale":"2023-10-11T11:19:02.769176+0000","last_undegraded":"2023-10-11T11:15:31.646946+0000","last_fullsized":"2023-10-11T11:15:31.646751+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:36.739637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:36.739637+0000","last_clean_scrub_stamp":"2023-10-11T10:43:36.739637+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T11:33:33.696447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":520,"reported_epoch":500,"state":"active+clean+remapped","last_fresh":"2023-10-11T11:19:03.429429+0000","last_change":"2023-10-11T11:15:33.660200+0000","last_active":"2023-10-11T11:19:03.429429+0000","last_peered":"2023-10-11T11:19:03.429429+0000","last_clean":"2023-10-11T11:19:03.429429+0000","last_became_active":"2023-10-11T11:15:33.659619+0000","last_became_peered":"2023-10-11T11:15:33.659619+0000","last_unstale":"2023-10-11T11:19:03.429429+0000","last_undegraded":"2023-10-11T11:19:03.429429+0000","last_fullsized":"2023-10-11T11:19:03.429429+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T17:21:16.866392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":517,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-10-11T11:19:02.775128+0000","last_change":"2023-10-11T11:15:31.670813+0000","last_active":"2023-10-11T11:19:02.775128+0000","last_peered":"2023-10-11T11:19:02.775128+0000","last_clean":"2023-10-11T11:15:27.638219+0000","last_became_active":"2023-10-11T11:15:31.670813+0000","last_became_peered":"2023-10-11T11:15:31.670813+0000","last_unstale":"2023-10-11T11:19:02.775128+0000","last_undegraded":"2023-10-11T11:19:02.775128+0000","last_fullsized":"2023-10-11T11:15:31.658785+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:41:50.872346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":516,"reported_epoch":499,"state":"active+undersized","last_fresh":"2023-10-11T11:19:00.631668+0000","last_change":"2023-10-11T11:15:31.666378+0000","last_active":"2023-10-11T11:19:00.631668+0000","last_peered":"2023-10-11T11:19:00.631668+0000","last_clean":"2023-10-11T11:15:27.638303+0000","last_became_active":"2023-10-11T11:15:31.666378+0000","last_became_peered":"2023-10-11T11:15:31.666378+0000","last_unstale":"2023-10-11T11:19:00.631668+0000","last_undegraded":"2023-10-11T11:19:00.631668+0000","last_fullsized":"2023-10-11T11:15:31.649186+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T15:35:19.331209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":516,"reported_epoch":499,"state":"active+undersized","last_fresh":"2023-10-11T11:19:00.636960+0000","last_change":"2023-10-11T11:15:31.679465+0000","last_active":"2023-10-11T11:19:00.636960+0000","last_peered":"2023-10-11T11:19:00.636960+0000","last_clean":"2023-10-11T11:15:27.638749+0000","last_became_active":"2023-10-11T11:15:31.679465+0000","last_became_peered":"2023-10-11T11:15:31.679465+0000","last_unstale":"2023-10-11T11:19:00.636960+0000","last_undegraded":"2023-10-11T11:19:00.636960+0000","last_fullsized":"2023-10-11T11:15:31.659135+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T16:03:42.656692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":11,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738813,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":785348,"kb_used_data":5812,"kb_used_omap":96,"kb_used_meta":779423,"kb_avail":93586492,"statfs":{"total":96636764160,"available":95832567808,"internally_reserved":0,"allocated":5951488,"data_stored":3106918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":98609,"internal_metadata":798129871},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 11:14:37 2023","interfaces":[{"interface":"back","average":{"1min":0.314,"5min":0.316,"15min":0.443},"min":{"1min":0.24399999999999999,"5min":0.14799999999999999,"15min":0.14599999999999999},"max":{"1min":0.48499999999999999,"5min":1.712,"15min":9.6400000000000006},"last":0.31},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.35499999999999998,"15min":0.41299999999999998},"min":{"1min":0.20899999999999999,"5min":0.17199999999999999,"15min":0.16400000000000001},"max":{"1min":3.8180000000000001,"5min":3.8180000000000001,"15min":6.3200000000000003},"last":0.26500000000000001}]},{"osd":1,"last update":"Wed Oct 11 11:19:40 2023","interfaces":[{"interface":"back","average":{"1min":0.29599999999999999,"5min":0.28499999999999998,"15min":0.32300000000000001},"min":{"1min":0.17699999999999999,"5min":0.154,"15min":0.154},"max":{"1min":0.40600000000000003,"5min":0.41999999999999998,"15min":2.9540000000000002},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.317,"5min":0.31,"15min":0.36199999999999999},"min":{"1min":0.20899999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.45000000000000001,"5min":0.49099999999999999,"15min":3.2719999999999998},"last":0.377}]}]},{"osd":0,"up_from":8,"seq":34359738756,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1484496,"kb_used_data":406792,"kb_used_omap":107,"kb_used_meta":1077588,"kb_avail":92887344,"statfs":{"total":96636764160,"available":95116640256,"internally_reserved":0,"allocated":416555008,"data_stored":413648921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110309,"internal_metadata":1103450395},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Oct 11 11:14:30 2023","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.35399999999999998,"15min":0.40600000000000003},"min":{"1min":0.18099999999999999,"5min":0.155,"15min":0.14899999999999999},"max":{"1min":1.4319999999999999,"5min":2.3460000000000001,"15min":6.593},"last":0.26200000000000001},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.39100000000000001,"15min":0.48199999999999998},"min":{"1min":0.189,"5min":0.189,"15min":0.183},"max":{"1min":2.2120000000000002,"5min":2.4089999999999998,"15min":6.5960000000000001},"last":0.316}]},{"osd":2,"last update":"Wed Oct 11 11:14:30 2023","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.35499999999999998,"15min":0.47099999999999997},"min":{"1min":0.20300000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":1.2490000000000001,"5min":2.665,"15min":8.4730000000000008},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.375,"15min":0.442},"min":{"1min":0.23200000000000001,"5min":0.20499999999999999,"15min":0.19600000000000001},"max":{"1min":1.28,"5min":2.6230000000000002,"15min":7.0129999999999999},"last":0.33600000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738814,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":826440,"kb_used_data":6392,"kb_used_omap":88,"kb_used_meta":819943,"kb_avail":93545400,"statfs":{"total":96636764160,"available":95790489600,"internally_reserved":0,"allocated":6545408,"data_stored":3697286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":90523,"internal_metadata":839622245},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 11:14:33 2023","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.42799999999999999,"15min":0.41099999999999998},"min":{"1min":0.151,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":1.2809999999999999,"5min":3.5739999999999998,"15min":8.2479999999999993},"last":0.223},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.48299999999999998,"15min":0.49199999999999999},"min":{"1min":0.20699999999999999,"5min":0.182,"15min":0.17999999999999999},"max":{"1min":1.323,"5min":3.6080000000000001,"15min":8.2870000000000008},"last":0.28399999999999997}]},{"osd":2,"last update":"Wed Oct 11 11:19:41 2023","interfaces":[{"interface":"back","average":{"1min":0.314,"5min":0.35399999999999998,"15min":0.35899999999999999},"min":{"1min":0.186,"5min":0.14699999999999999,"15min":0.14699999999999999},"max":{"1min":0.45500000000000002,"5min":7.4829999999999997,"15min":10.617000000000001},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.39400000000000002,"15min":0.35099999999999998},"min":{"1min":0.21199999999999999,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.49099999999999999,"5min":7.5350000000000001,"15min":7.5350000000000001},"last":0.29599999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-11T11:20:28.606 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-10-11T11:20:28.922 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T11:20:28.924 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2023-10-11T11:20:28.936 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":1605,"stamp":"2023-10-11T11:20:28.664942+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":11,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":107,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3096284,"kb_used_data":418996,"kb_used_omap":292,"kb_used_meta":2676955,"kb_avail":280019236,"statfs":{"total":289910292480,"available":286739697664,"internally_reserved":0,"allocated":429051904,"data_stored":420453125,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299441,"internal_metadata":2741202511},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003072"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":517,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-10-11T11:19:02.772828+0000","last_change":"2023-10-11T11:15:31.661747+0000","last_active":"2023-10-11T11:19:02.772828+0000","last_peered":"2023-10-11T11:19:02.772828+0000","last_clean":"2023-10-11T11:15:27.638215+0000","last_became_active":"2023-10-11T11:15:31.661747+0000","last_became_peered":"2023-10-11T11:15:31.661747+0000","last_unstale":"2023-10-11T11:19:02.772828+0000","last_undegraded":"2023-10-11T11:19:02.772828+0000","last_fullsized":"2023-10-11T11:15:31.649336+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:15:25.047039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":29,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-10-11T11:19:02.774213+0000","last_change":"2023-10-11T11:15:31.671363+0000","last_active":"2023-10-11T11:19:02.774213+0000","last_peered":"2023-10-11T11:19:02.774213+0000","last_clean":"2023-10-11T10:43:40.751300+0000","last_became_active":"2023-10-11T11:15:31.671363+0000","last_became_peered":"2023-10-11T11:15:31.671363+0000","last_unstale":"2023-10-11T11:19:02.774213+0000","last_undegraded":"2023-10-11T11:19:02.774213+0000","last_fullsized":"2023-10-11T11:15:31.652001+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:01:53.884964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":512,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-10-11T11:19:03.429640+0000","last_change":"2023-10-11T10:43:41.772781+0000","last_active":"2023-10-11T11:19:03.429640+0000","last_peered":"2023-10-11T11:19:03.429640+0000","last_clean":"2023-10-11T11:19:03.429640+0000","last_became_active":"2023-10-11T10:43:41.772443+0000","last_became_peered":"2023-10-11T10:43:41.772443+0000","last_unstale":"2023-10-11T11:19:03.429640+0000","last_undegraded":"2023-10-11T11:19:03.429640+0000","last_fullsized":"2023-10-11T11:19:03.429640+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T14:15:30.547284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":513,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-10-11T11:19:02.772211+0000","last_change":"2023-10-11T10:43:41.772615+0000","last_active":"2023-10-11T11:19:02.772211+0000","last_peered":"2023-10-11T11:19:02.772211+0000","last_clean":"2023-10-11T11:19:02.772211+0000","last_became_active":"2023-10-11T10:43:41.772290+0000","last_became_peered":"2023-10-11T10:43:41.772290+0000","last_unstale":"2023-10-11T11:19:02.772211+0000","last_undegraded":"2023-10-11T11:19:02.772211+0000","last_fullsized":"2023-10-11T11:19:02.772211+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:26:06.909963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":648,"reported_epoch":500,"state":"active+undersized+degraded","last_fresh":"2023-10-11T11:19:02.769176+0000","last_change":"2023-10-11T11:15:31.654022+0000","last_active":"2023-10-11T11:19:02.769176+0000","last_peered":"2023-10-11T11:19:02.769176+0000","last_clean":"2023-10-11T11:15:27.636884+0000","last_became_active":"2023-10-11T11:15:31.654022+0000","last_became_peered":"2023-10-11T11:15:31.654022+0000","last_unstale":"2023-10-11T11:19:02.769176+0000","last_undegraded":"2023-10-11T11:15:31.646946+0000","last_fullsized":"2023-10-11T11:15:31.646751+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:36.739637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:36.739637+0000","last_clean_scrub_stamp":"2023-10-11T10:43:36.739637+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T11:33:33.696447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":520,"reported_epoch":500,"state":"active+clean+remapped","last_fresh":"2023-10-11T11:19:03.429429+0000","last_change":"2023-10-11T11:15:33.660200+0000","last_active":"2023-10-11T11:19:03.429429+0000","last_peered":"2023-10-11T11:19:03.429429+0000","last_clean":"2023-10-11T11:19:03.429429+0000","last_became_active":"2023-10-11T11:15:33.659619+0000","last_became_peered":"2023-10-11T11:15:33.659619+0000","last_unstale":"2023-10-11T11:19:03.429429+0000","last_undegraded":"2023-10-11T11:19:03.429429+0000","last_fullsized":"2023-10-11T11:19:03.429429+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T17:21:16.866392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":517,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-10-11T11:19:02.775128+0000","last_change":"2023-10-11T11:15:31.670813+0000","last_active":"2023-10-11T11:19:02.775128+0000","last_peered":"2023-10-11T11:19:02.775128+0000","last_clean":"2023-10-11T11:15:27.638219+0000","last_became_active":"2023-10-11T11:15:31.670813+0000","last_became_peered":"2023-10-11T11:15:31.670813+0000","last_unstale":"2023-10-11T11:19:02.775128+0000","last_undegraded":"2023-10-11T11:19:02.775128+0000","last_fullsized":"2023-10-11T11:15:31.658785+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:41:50.872346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":516,"reported_epoch":499,"state":"active+undersized","last_fresh":"2023-10-11T11:19:00.631668+0000","last_change":"2023-10-11T11:15:31.666378+0000","last_active":"2023-10-11T11:19:00.631668+0000","last_peered":"2023-10-11T11:19:00.631668+0000","last_clean":"2023-10-11T11:15:27.638303+0000","last_became_active":"2023-10-11T11:15:31.666378+0000","last_became_peered":"2023-10-11T11:15:31.666378+0000","last_unstale":"2023-10-11T11:19:00.631668+0000","last_undegraded":"2023-10-11T11:19:00.631668+0000","last_fullsized":"2023-10-11T11:15:31.649186+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T15:35:19.331209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":516,"reported_epoch":499,"state":"active+undersized","last_fresh":"2023-10-11T11:19:00.636960+0000","last_change":"2023-10-11T11:15:31.679465+0000","last_active":"2023-10-11T11:19:00.636960+0000","last_peered":"2023-10-11T11:19:00.636960+0000","last_clean":"2023-10-11T11:15:27.638749+0000","last_became_active":"2023-10-11T11:15:31.679465+0000","last_became_peered":"2023-10-11T11:15:31.679465+0000","last_unstale":"2023-10-11T11:19:00.636960+0000","last_undegraded":"2023-10-11T11:19:00.636960+0000","last_fullsized":"2023-10-11T11:15:31.659135+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T16:03:42.656692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":11,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738813,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":785348,"kb_used_data":5812,"kb_used_omap":96,"kb_used_meta":779423,"kb_avail":93586492,"statfs":{"total":96636764160,"available":95832567808,"internally_reserved":0,"allocated":5951488,"data_stored":3106918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":98609,"internal_metadata":798129871},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 11:14:37 2023","interfaces":[{"interface":"back","average":{"1min":0.314,"5min":0.316,"15min":0.443},"min":{"1min":0.24399999999999999,"5min":0.14799999999999999,"15min":0.14599999999999999},"max":{"1min":0.48499999999999999,"5min":1.712,"15min":9.6400000000000006},"last":0.31},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.35499999999999998,"15min":0.41299999999999998},"min":{"1min":0.20899999999999999,"5min":0.17199999999999999,"15min":0.16400000000000001},"max":{"1min":3.8180000000000001,"5min":3.8180000000000001,"15min":6.3200000000000003},"last":0.26500000000000001}]},{"osd":1,"last update":"Wed Oct 11 11:19:40 2023","interfaces":[{"interface":"back","average":{"1min":0.29599999999999999,"5min":0.28499999999999998,"15min":0.32300000000000001},"min":{"1min":0.17699999999999999,"5min":0.154,"15min":0.154},"max":{"1min":0.40600000000000003,"5min":0.41999999999999998,"15min":2.9540000000000002},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.317,"5min":0.31,"15min":0.36199999999999999},"min":{"1min":0.20899999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.45000000000000001,"5min":0.49099999999999999,"15min":3.2719999999999998},"last":0.377}]}]},{"osd":0,"up_from":8,"seq":34359738756,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1484496,"kb_used_data":406792,"kb_used_omap":107,"kb_used_meta":1077588,"kb_avail":92887344,"statfs":{"total":96636764160,"available":95116640256,"internally_reserved":0,"allocated":416555008,"data_stored":413648921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110309,"internal_metadata":1103450395},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Oct 11 11:14:30 2023","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.35399999999999998,"15min":0.40600000000000003},"min":{"1min":0.18099999999999999,"5min":0.155,"15min":0.14899999999999999},"max":{"1min":1.4319999999999999,"5min":2.3460000000000001,"15min":6.593},"last":0.26200000000000001},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.39100000000000001,"15min":0.48199999999999998},"min":{"1min":0.189,"5min":0.189,"15min":0.183},"max":{"1min":2.2120000000000002,"5min":2.4089999999999998,"15min":6.5960000000000001},"last":0.316}]},{"osd":2,"last update":"Wed Oct 11 11:14:30 2023","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.35499999999999998,"15min":0.47099999999999997},"min":{"1min":0.20300000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":1.2490000000000001,"5min":2.665,"15min":8.4730000000000008},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.375,"15min":0.442},"min":{"1min":0.23200000000000001,"5min":0.20499999999999999,"15min":0.19600000000000001},"max":{"1min":1.28,"5min":2.6230000000000002,"15min":7.0129999999999999},"last":0.33600000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738814,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":826440,"kb_used_data":6392,"kb_used_omap":88,"kb_used_meta":819943,"kb_avail":93545400,"statfs":{"total":96636764160,"available":95790489600,"internally_reserved":0,"allocated":6545408,"data_stored":3697286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":90523,"internal_metadata":839622245},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 11:14:33 2023","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.42799999999999999,"15min":0.41099999999999998},"min":{"1min":0.151,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":1.2809999999999999,"5min":3.5739999999999998,"15min":8.2479999999999993},"last":0.223},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.48299999999999998,"15min":0.49199999999999999},"min":{"1min":0.20699999999999999,"5min":0.182,"15min":0.17999999999999999},"max":{"1min":1.323,"5min":3.6080000000000001,"15min":8.2870000000000008},"last":0.28399999999999997}]},{"osd":2,"last update":"Wed Oct 11 11:19:41 2023","interfaces":[{"interface":"back","average":{"1min":0.314,"5min":0.35399999999999998,"15min":0.35899999999999999},"min":{"1min":0.186,"5min":0.14699999999999999,"15min":0.14699999999999999},"max":{"1min":0.45500000000000002,"5min":7.4829999999999997,"15min":10.617000000000001},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.39400000000000002,"15min":0.35099999999999998},"min":{"1min":0.21199999999999999,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.49099999999999999,"5min":7.5350000000000001,"15min":7.5350000000000001},"last":0.29599999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-11T11:20:28.937 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-10-11T11:20:28.937 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 517, 'reported_epoch': 500, 'state': 'active+undersized', 'last_fresh': '2023-10-11T11:19:02.772828+0000', 'last_change': '2023-10-11T11:15:31.661747+0000', 'last_active': '2023-10-11T11:19:02.772828+0000', 'last_peered': '2023-10-11T11:19:02.772828+0000', 'last_clean': '2023-10-11T11:15:27.638215+0000', 'last_became_active': '2023-10-11T11:15:31.661747+0000', 'last_became_peered': '2023-10-11T11:15:31.661747+0000', 'last_unstale': '2023-10-11T11:19:02.772828+0000', 'last_undegraded': '2023-10-11T11:19:02.772828+0000', 'last_fullsized': '2023-10-11T11:15:31.649336+0000', 'mapping_epoch': 480, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_clean_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-12T22:15:25.047039+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-10-11T11:20:28.937 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-10-11T11:20:28.938 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 29, 'reported_epoch': 500, 'state': 'active+undersized', 'last_fresh': '2023-10-11T11:19:02.774213+0000', 'last_change': '2023-10-11T11:15:31.671363+0000', 'last_active': '2023-10-11T11:19:02.774213+0000', 'last_peered': '2023-10-11T11:19:02.774213+0000', 'last_clean': '2023-10-11T10:43:40.751300+0000', 'last_became_active': '2023-10-11T11:15:31.671363+0000', 'last_became_peered': '2023-10-11T11:15:31.671363+0000', 'last_unstale': '2023-10-11T11:19:02.774213+0000', 'last_undegraded': '2023-10-11T11:19:02.774213+0000', 'last_fullsized': '2023-10-11T11:15:31.652001+0000', 'mapping_epoch': 480, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_clean_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-12T21:01:53.884964+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-10-11T11:20:28.938 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-10-11T11:20:28.938 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "11'97", 'reported_seq': 648, 'reported_epoch': 500, 'state': 'active+undersized+degraded', 'last_fresh': '2023-10-11T11:19:02.769176+0000', 'last_change': '2023-10-11T11:15:31.654022+0000', 'last_active': '2023-10-11T11:19:02.769176+0000', 'last_peered': '2023-10-11T11:19:02.769176+0000', 'last_clean': '2023-10-11T11:15:27.636884+0000', 'last_became_active': '2023-10-11T11:15:31.654022+0000', 'last_became_peered': '2023-10-11T11:15:31.654022+0000', 'last_unstale': '2023-10-11T11:19:02.769176+0000', 'last_undegraded': '2023-10-11T11:15:31.646946+0000', 'last_fullsized': '2023-10-11T11:15:31.646751+0000', 'mapping_epoch': 480, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-11T10:43:36.739637+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-11T10:43:36.739637+0000', 'last_clean_scrub_stamp': '2023-10-11T10:43:36.739637+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-12T11:33:33.696447+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-10-11T11:20:28.939 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.0 is not active+clean 2023-10-11T11:20:28.939 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 520, 'reported_epoch': 500, 'state': 'active+clean+remapped', 'last_fresh': '2023-10-11T11:19:03.429429+0000', 'last_change': '2023-10-11T11:15:33.660200+0000', 'last_active': '2023-10-11T11:19:03.429429+0000', 'last_peered': '2023-10-11T11:19:03.429429+0000', 'last_clean': '2023-10-11T11:19:03.429429+0000', 'last_became_active': '2023-10-11T11:15:33.659619+0000', 'last_became_peered': '2023-10-11T11:15:33.659619+0000', 'last_unstale': '2023-10-11T11:19:03.429429+0000', 'last_undegraded': '2023-10-11T11:19:03.429429+0000', 'last_fullsized': '2023-10-11T11:19:03.429429+0000', 'mapping_epoch': 482, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 483, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_clean_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-12T17:21:16.866392+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2, 1], 'avail_no_missing': ['2', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-10-11T11:20:28.939 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-10-11T11:20:28.939 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 517, 'reported_epoch': 500, 'state': 'active+undersized', 'last_fresh': '2023-10-11T11:19:02.775128+0000', 'last_change': '2023-10-11T11:15:31.670813+0000', 'last_active': '2023-10-11T11:19:02.775128+0000', 'last_peered': '2023-10-11T11:19:02.775128+0000', 'last_clean': '2023-10-11T11:15:27.638219+0000', 'last_became_active': '2023-10-11T11:15:31.670813+0000', 'last_became_peered': '2023-10-11T11:15:31.670813+0000', 'last_unstale': '2023-10-11T11:19:02.775128+0000', 'last_undegraded': '2023-10-11T11:19:02.775128+0000', 'last_fullsized': '2023-10-11T11:15:31.658785+0000', 'mapping_epoch': 480, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_clean_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-12T21:41:50.872346+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-10-11T11:20:28.940 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-10-11T11:20:28.940 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 516, 'reported_epoch': 499, 'state': 'active+undersized', 'last_fresh': '2023-10-11T11:19:00.631668+0000', 'last_change': '2023-10-11T11:15:31.666378+0000', 'last_active': '2023-10-11T11:19:00.631668+0000', 'last_peered': '2023-10-11T11:19:00.631668+0000', 'last_clean': '2023-10-11T11:15:27.638303+0000', 'last_became_active': '2023-10-11T11:15:31.666378+0000', 'last_became_peered': '2023-10-11T11:15:31.666378+0000', 'last_unstale': '2023-10-11T11:19:00.631668+0000', 'last_undegraded': '2023-10-11T11:19:00.631668+0000', 'last_fullsized': '2023-10-11T11:15:31.649186+0000', 'mapping_epoch': 480, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_clean_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-12T15:35:19.331209+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-10-11T11:20:28.940 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-10-11T11:20:28.940 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 516, 'reported_epoch': 499, 'state': 'active+undersized', 'last_fresh': '2023-10-11T11:19:00.636960+0000', 'last_change': '2023-10-11T11:15:31.679465+0000', 'last_active': '2023-10-11T11:19:00.636960+0000', 'last_peered': '2023-10-11T11:19:00.636960+0000', 'last_clean': '2023-10-11T11:15:27.638749+0000', 'last_became_active': '2023-10-11T11:15:31.679465+0000', 'last_became_peered': '2023-10-11T11:15:31.679465+0000', 'last_unstale': '2023-10-11T11:19:00.636960+0000', 'last_undegraded': '2023-10-11T11:19:00.636960+0000', 'last_fullsized': '2023-10-11T11:15:31.659135+0000', 'mapping_epoch': 480, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_clean_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-12T16:03:42.656692+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-10-11T11:20:28.941 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-10-11T11:20:29.335 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T11:20:29.336 INFO:teuthology.orchestra.run.smithi111.stdout:{"fsid":"022ce6f2-9a5b-45ad-8b77-a1332d92f228","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2224,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":500,"num_osds":3,"num_up_osds":2,"osd_up_since":1697022930,"num_in_osds":3,"osd_in_since":1697021005,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":2},{"state_name":"active+undersized+degraded","count":1},{"state_name":"active+clean+remapped","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3170594816,"bytes_avail":286739697664,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":36,"modified":"2023-10-11T11:19:04.644408+0000","services":{}},"progress_events":{}} 2023-10-11T11:20:29.350 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-10-11T11:20:29.350 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-10-11T11:20:29.369 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~297s 2023-10-11T11:20:29.666 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T11:20:29.668 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2023-10-11T11:20:29.680 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":1605,"stamp":"2023-10-11T11:20:28.664942+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":11,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":107,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3096284,"kb_used_data":418996,"kb_used_omap":292,"kb_used_meta":2676955,"kb_avail":280019236,"statfs":{"total":289910292480,"available":286739697664,"internally_reserved":0,"allocated":429051904,"data_stored":420453125,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299441,"internal_metadata":2741202511},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003072"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":517,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-10-11T11:19:02.772828+0000","last_change":"2023-10-11T11:15:31.661747+0000","last_active":"2023-10-11T11:19:02.772828+0000","last_peered":"2023-10-11T11:19:02.772828+0000","last_clean":"2023-10-11T11:15:27.638215+0000","last_became_active":"2023-10-11T11:15:31.661747+0000","last_became_peered":"2023-10-11T11:15:31.661747+0000","last_unstale":"2023-10-11T11:19:02.772828+0000","last_undegraded":"2023-10-11T11:19:02.772828+0000","last_fullsized":"2023-10-11T11:15:31.649336+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:15:25.047039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":29,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-10-11T11:19:02.774213+0000","last_change":"2023-10-11T11:15:31.671363+0000","last_active":"2023-10-11T11:19:02.774213+0000","last_peered":"2023-10-11T11:19:02.774213+0000","last_clean":"2023-10-11T10:43:40.751300+0000","last_became_active":"2023-10-11T11:15:31.671363+0000","last_became_peered":"2023-10-11T11:15:31.671363+0000","last_unstale":"2023-10-11T11:19:02.774213+0000","last_undegraded":"2023-10-11T11:19:02.774213+0000","last_fullsized":"2023-10-11T11:15:31.652001+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:01:53.884964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":512,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-10-11T11:19:03.429640+0000","last_change":"2023-10-11T10:43:41.772781+0000","last_active":"2023-10-11T11:19:03.429640+0000","last_peered":"2023-10-11T11:19:03.429640+0000","last_clean":"2023-10-11T11:19:03.429640+0000","last_became_active":"2023-10-11T10:43:41.772443+0000","last_became_peered":"2023-10-11T10:43:41.772443+0000","last_unstale":"2023-10-11T11:19:03.429640+0000","last_undegraded":"2023-10-11T11:19:03.429640+0000","last_fullsized":"2023-10-11T11:19:03.429640+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T14:15:30.547284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":513,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-10-11T11:19:02.772211+0000","last_change":"2023-10-11T10:43:41.772615+0000","last_active":"2023-10-11T11:19:02.772211+0000","last_peered":"2023-10-11T11:19:02.772211+0000","last_clean":"2023-10-11T11:19:02.772211+0000","last_became_active":"2023-10-11T10:43:41.772290+0000","last_became_peered":"2023-10-11T10:43:41.772290+0000","last_unstale":"2023-10-11T11:19:02.772211+0000","last_undegraded":"2023-10-11T11:19:02.772211+0000","last_fullsized":"2023-10-11T11:19:02.772211+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:26:06.909963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":648,"reported_epoch":500,"state":"active+undersized+degraded","last_fresh":"2023-10-11T11:19:02.769176+0000","last_change":"2023-10-11T11:15:31.654022+0000","last_active":"2023-10-11T11:19:02.769176+0000","last_peered":"2023-10-11T11:19:02.769176+0000","last_clean":"2023-10-11T11:15:27.636884+0000","last_became_active":"2023-10-11T11:15:31.654022+0000","last_became_peered":"2023-10-11T11:15:31.654022+0000","last_unstale":"2023-10-11T11:19:02.769176+0000","last_undegraded":"2023-10-11T11:15:31.646946+0000","last_fullsized":"2023-10-11T11:15:31.646751+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:36.739637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:36.739637+0000","last_clean_scrub_stamp":"2023-10-11T10:43:36.739637+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T11:33:33.696447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":520,"reported_epoch":500,"state":"active+clean+remapped","last_fresh":"2023-10-11T11:19:03.429429+0000","last_change":"2023-10-11T11:15:33.660200+0000","last_active":"2023-10-11T11:19:03.429429+0000","last_peered":"2023-10-11T11:19:03.429429+0000","last_clean":"2023-10-11T11:19:03.429429+0000","last_became_active":"2023-10-11T11:15:33.659619+0000","last_became_peered":"2023-10-11T11:15:33.659619+0000","last_unstale":"2023-10-11T11:19:03.429429+0000","last_undegraded":"2023-10-11T11:19:03.429429+0000","last_fullsized":"2023-10-11T11:19:03.429429+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T17:21:16.866392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":517,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-10-11T11:19:02.775128+0000","last_change":"2023-10-11T11:15:31.670813+0000","last_active":"2023-10-11T11:19:02.775128+0000","last_peered":"2023-10-11T11:19:02.775128+0000","last_clean":"2023-10-11T11:15:27.638219+0000","last_became_active":"2023-10-11T11:15:31.670813+0000","last_became_peered":"2023-10-11T11:15:31.670813+0000","last_unstale":"2023-10-11T11:19:02.775128+0000","last_undegraded":"2023-10-11T11:19:02.775128+0000","last_fullsized":"2023-10-11T11:15:31.658785+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:41:50.872346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":516,"reported_epoch":499,"state":"active+undersized","last_fresh":"2023-10-11T11:19:00.631668+0000","last_change":"2023-10-11T11:15:31.666378+0000","last_active":"2023-10-11T11:19:00.631668+0000","last_peered":"2023-10-11T11:19:00.631668+0000","last_clean":"2023-10-11T11:15:27.638303+0000","last_became_active":"2023-10-11T11:15:31.666378+0000","last_became_peered":"2023-10-11T11:15:31.666378+0000","last_unstale":"2023-10-11T11:19:00.631668+0000","last_undegraded":"2023-10-11T11:19:00.631668+0000","last_fullsized":"2023-10-11T11:15:31.649186+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T15:35:19.331209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":516,"reported_epoch":499,"state":"active+undersized","last_fresh":"2023-10-11T11:19:00.636960+0000","last_change":"2023-10-11T11:15:31.679465+0000","last_active":"2023-10-11T11:19:00.636960+0000","last_peered":"2023-10-11T11:19:00.636960+0000","last_clean":"2023-10-11T11:15:27.638749+0000","last_became_active":"2023-10-11T11:15:31.679465+0000","last_became_peered":"2023-10-11T11:15:31.679465+0000","last_unstale":"2023-10-11T11:19:00.636960+0000","last_undegraded":"2023-10-11T11:19:00.636960+0000","last_fullsized":"2023-10-11T11:15:31.659135+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T16:03:42.656692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":11,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738813,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":785348,"kb_used_data":5812,"kb_used_omap":96,"kb_used_meta":779423,"kb_avail":93586492,"statfs":{"total":96636764160,"available":95832567808,"internally_reserved":0,"allocated":5951488,"data_stored":3106918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":98609,"internal_metadata":798129871},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 11:14:37 2023","interfaces":[{"interface":"back","average":{"1min":0.314,"5min":0.316,"15min":0.443},"min":{"1min":0.24399999999999999,"5min":0.14799999999999999,"15min":0.14599999999999999},"max":{"1min":0.48499999999999999,"5min":1.712,"15min":9.6400000000000006},"last":0.31},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.35499999999999998,"15min":0.41299999999999998},"min":{"1min":0.20899999999999999,"5min":0.17199999999999999,"15min":0.16400000000000001},"max":{"1min":3.8180000000000001,"5min":3.8180000000000001,"15min":6.3200000000000003},"last":0.26500000000000001}]},{"osd":1,"last update":"Wed Oct 11 11:19:40 2023","interfaces":[{"interface":"back","average":{"1min":0.29599999999999999,"5min":0.28499999999999998,"15min":0.32300000000000001},"min":{"1min":0.17699999999999999,"5min":0.154,"15min":0.154},"max":{"1min":0.40600000000000003,"5min":0.41999999999999998,"15min":2.9540000000000002},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.317,"5min":0.31,"15min":0.36199999999999999},"min":{"1min":0.20899999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.45000000000000001,"5min":0.49099999999999999,"15min":3.2719999999999998},"last":0.377}]}]},{"osd":0,"up_from":8,"seq":34359738756,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1484496,"kb_used_data":406792,"kb_used_omap":107,"kb_used_meta":1077588,"kb_avail":92887344,"statfs":{"total":96636764160,"available":95116640256,"internally_reserved":0,"allocated":416555008,"data_stored":413648921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110309,"internal_metadata":1103450395},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Oct 11 11:14:30 2023","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.35399999999999998,"15min":0.40600000000000003},"min":{"1min":0.18099999999999999,"5min":0.155,"15min":0.14899999999999999},"max":{"1min":1.4319999999999999,"5min":2.3460000000000001,"15min":6.593},"last":0.26200000000000001},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.39100000000000001,"15min":0.48199999999999998},"min":{"1min":0.189,"5min":0.189,"15min":0.183},"max":{"1min":2.2120000000000002,"5min":2.4089999999999998,"15min":6.5960000000000001},"last":0.316}]},{"osd":2,"last update":"Wed Oct 11 11:14:30 2023","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.35499999999999998,"15min":0.47099999999999997},"min":{"1min":0.20300000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":1.2490000000000001,"5min":2.665,"15min":8.4730000000000008},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.375,"15min":0.442},"min":{"1min":0.23200000000000001,"5min":0.20499999999999999,"15min":0.19600000000000001},"max":{"1min":1.28,"5min":2.6230000000000002,"15min":7.0129999999999999},"last":0.33600000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738814,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":826440,"kb_used_data":6392,"kb_used_omap":88,"kb_used_meta":819943,"kb_avail":93545400,"statfs":{"total":96636764160,"available":95790489600,"internally_reserved":0,"allocated":6545408,"data_stored":3697286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":90523,"internal_metadata":839622245},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 11:14:33 2023","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.42799999999999999,"15min":0.41099999999999998},"min":{"1min":0.151,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":1.2809999999999999,"5min":3.5739999999999998,"15min":8.2479999999999993},"last":0.223},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.48299999999999998,"15min":0.49199999999999999},"min":{"1min":0.20699999999999999,"5min":0.182,"15min":0.17999999999999999},"max":{"1min":1.323,"5min":3.6080000000000001,"15min":8.2870000000000008},"last":0.28399999999999997}]},{"osd":2,"last update":"Wed Oct 11 11:19:41 2023","interfaces":[{"interface":"back","average":{"1min":0.314,"5min":0.35399999999999998,"15min":0.35899999999999999},"min":{"1min":0.186,"5min":0.14699999999999999,"15min":0.14699999999999999},"max":{"1min":0.45500000000000002,"5min":7.4829999999999997,"15min":10.617000000000001},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.39400000000000002,"15min":0.35099999999999998},"min":{"1min":0.21199999999999999,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.49099999999999999,"5min":7.5350000000000001,"15min":7.5350000000000001},"last":0.29599999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-11T11:20:32.682 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-10-11T11:20:33.013 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T11:20:33.014 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2023-10-11T11:20:33.026 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":1607,"stamp":"2023-10-11T11:20:32.666112+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":11,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":107,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3096284,"kb_used_data":418996,"kb_used_omap":292,"kb_used_meta":2676955,"kb_avail":280019236,"statfs":{"total":289910292480,"available":286739697664,"internally_reserved":0,"allocated":429051904,"data_stored":420453125,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299441,"internal_metadata":2741202511},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003313"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":517,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-10-11T11:19:02.772828+0000","last_change":"2023-10-11T11:15:31.661747+0000","last_active":"2023-10-11T11:19:02.772828+0000","last_peered":"2023-10-11T11:19:02.772828+0000","last_clean":"2023-10-11T11:15:27.638215+0000","last_became_active":"2023-10-11T11:15:31.661747+0000","last_became_peered":"2023-10-11T11:15:31.661747+0000","last_unstale":"2023-10-11T11:19:02.772828+0000","last_undegraded":"2023-10-11T11:19:02.772828+0000","last_fullsized":"2023-10-11T11:15:31.649336+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:15:25.047039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":29,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-10-11T11:19:02.774213+0000","last_change":"2023-10-11T11:15:31.671363+0000","last_active":"2023-10-11T11:19:02.774213+0000","last_peered":"2023-10-11T11:19:02.774213+0000","last_clean":"2023-10-11T10:43:40.751300+0000","last_became_active":"2023-10-11T11:15:31.671363+0000","last_became_peered":"2023-10-11T11:15:31.671363+0000","last_unstale":"2023-10-11T11:19:02.774213+0000","last_undegraded":"2023-10-11T11:19:02.774213+0000","last_fullsized":"2023-10-11T11:15:31.652001+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:01:53.884964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":512,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-10-11T11:19:03.429640+0000","last_change":"2023-10-11T10:43:41.772781+0000","last_active":"2023-10-11T11:19:03.429640+0000","last_peered":"2023-10-11T11:19:03.429640+0000","last_clean":"2023-10-11T11:19:03.429640+0000","last_became_active":"2023-10-11T10:43:41.772443+0000","last_became_peered":"2023-10-11T10:43:41.772443+0000","last_unstale":"2023-10-11T11:19:03.429640+0000","last_undegraded":"2023-10-11T11:19:03.429640+0000","last_fullsized":"2023-10-11T11:19:03.429640+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T14:15:30.547284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":513,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-10-11T11:19:02.772211+0000","last_change":"2023-10-11T10:43:41.772615+0000","last_active":"2023-10-11T11:19:02.772211+0000","last_peered":"2023-10-11T11:19:02.772211+0000","last_clean":"2023-10-11T11:19:02.772211+0000","last_became_active":"2023-10-11T10:43:41.772290+0000","last_became_peered":"2023-10-11T10:43:41.772290+0000","last_unstale":"2023-10-11T11:19:02.772211+0000","last_undegraded":"2023-10-11T11:19:02.772211+0000","last_fullsized":"2023-10-11T11:19:02.772211+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:26:06.909963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":648,"reported_epoch":500,"state":"active+undersized+degraded","last_fresh":"2023-10-11T11:19:02.769176+0000","last_change":"2023-10-11T11:15:31.654022+0000","last_active":"2023-10-11T11:19:02.769176+0000","last_peered":"2023-10-11T11:19:02.769176+0000","last_clean":"2023-10-11T11:15:27.636884+0000","last_became_active":"2023-10-11T11:15:31.654022+0000","last_became_peered":"2023-10-11T11:15:31.654022+0000","last_unstale":"2023-10-11T11:19:02.769176+0000","last_undegraded":"2023-10-11T11:15:31.646946+0000","last_fullsized":"2023-10-11T11:15:31.646751+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:36.739637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:36.739637+0000","last_clean_scrub_stamp":"2023-10-11T10:43:36.739637+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T11:33:33.696447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":520,"reported_epoch":500,"state":"active+clean+remapped","last_fresh":"2023-10-11T11:19:03.429429+0000","last_change":"2023-10-11T11:15:33.660200+0000","last_active":"2023-10-11T11:19:03.429429+0000","last_peered":"2023-10-11T11:19:03.429429+0000","last_clean":"2023-10-11T11:19:03.429429+0000","last_became_active":"2023-10-11T11:15:33.659619+0000","last_became_peered":"2023-10-11T11:15:33.659619+0000","last_unstale":"2023-10-11T11:19:03.429429+0000","last_undegraded":"2023-10-11T11:19:03.429429+0000","last_fullsized":"2023-10-11T11:19:03.429429+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T17:21:16.866392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":517,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-10-11T11:19:02.775128+0000","last_change":"2023-10-11T11:15:31.670813+0000","last_active":"2023-10-11T11:19:02.775128+0000","last_peered":"2023-10-11T11:19:02.775128+0000","last_clean":"2023-10-11T11:15:27.638219+0000","last_became_active":"2023-10-11T11:15:31.670813+0000","last_became_peered":"2023-10-11T11:15:31.670813+0000","last_unstale":"2023-10-11T11:19:02.775128+0000","last_undegraded":"2023-10-11T11:19:02.775128+0000","last_fullsized":"2023-10-11T11:15:31.658785+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:41:50.872346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":517,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-10-11T11:19:02.772818+0000","last_change":"2023-10-11T11:15:31.666378+0000","last_active":"2023-10-11T11:19:02.772818+0000","last_peered":"2023-10-11T11:19:02.772818+0000","last_clean":"2023-10-11T11:15:27.638303+0000","last_became_active":"2023-10-11T11:15:31.666378+0000","last_became_peered":"2023-10-11T11:15:31.666378+0000","last_unstale":"2023-10-11T11:19:02.772818+0000","last_undegraded":"2023-10-11T11:19:02.772818+0000","last_fullsized":"2023-10-11T11:15:31.649186+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T15:35:19.331209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":516,"reported_epoch":499,"state":"active+undersized","last_fresh":"2023-10-11T11:19:00.636960+0000","last_change":"2023-10-11T11:15:31.679465+0000","last_active":"2023-10-11T11:19:00.636960+0000","last_peered":"2023-10-11T11:19:00.636960+0000","last_clean":"2023-10-11T11:15:27.638749+0000","last_became_active":"2023-10-11T11:15:31.679465+0000","last_became_peered":"2023-10-11T11:15:31.679465+0000","last_unstale":"2023-10-11T11:19:00.636960+0000","last_undegraded":"2023-10-11T11:19:00.636960+0000","last_fullsized":"2023-10-11T11:15:31.659135+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T16:03:42.656692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":11,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738814,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":785348,"kb_used_data":5812,"kb_used_omap":96,"kb_used_meta":779423,"kb_avail":93586492,"statfs":{"total":96636764160,"available":95832567808,"internally_reserved":0,"allocated":5951488,"data_stored":3106918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":98609,"internal_metadata":798129871},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 11:14:37 2023","interfaces":[{"interface":"back","average":{"1min":0.314,"5min":0.316,"15min":0.443},"min":{"1min":0.24399999999999999,"5min":0.14799999999999999,"15min":0.14599999999999999},"max":{"1min":0.48499999999999999,"5min":1.712,"15min":9.6400000000000006},"last":0.31},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.35499999999999998,"15min":0.41299999999999998},"min":{"1min":0.20899999999999999,"5min":0.17199999999999999,"15min":0.16400000000000001},"max":{"1min":3.8180000000000001,"5min":3.8180000000000001,"15min":6.3200000000000003},"last":0.26500000000000001}]},{"osd":1,"last update":"Wed Oct 11 11:19:40 2023","interfaces":[{"interface":"back","average":{"1min":0.29599999999999999,"5min":0.28499999999999998,"15min":0.32300000000000001},"min":{"1min":0.17699999999999999,"5min":0.154,"15min":0.154},"max":{"1min":0.40600000000000003,"5min":0.41999999999999998,"15min":2.9540000000000002},"last":0.22800000000000001},{"interface":"front","average":{"1min":0.317,"5min":0.31,"15min":0.36199999999999999},"min":{"1min":0.20899999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.45000000000000001,"5min":0.49099999999999999,"15min":3.2719999999999998},"last":0.27300000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738756,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1484496,"kb_used_data":406792,"kb_used_omap":107,"kb_used_meta":1077588,"kb_avail":92887344,"statfs":{"total":96636764160,"available":95116640256,"internally_reserved":0,"allocated":416555008,"data_stored":413648921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110309,"internal_metadata":1103450395},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Oct 11 11:14:30 2023","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.35399999999999998,"15min":0.40600000000000003},"min":{"1min":0.18099999999999999,"5min":0.155,"15min":0.14899999999999999},"max":{"1min":1.4319999999999999,"5min":2.3460000000000001,"15min":6.593},"last":0.26200000000000001},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.39100000000000001,"15min":0.48199999999999998},"min":{"1min":0.189,"5min":0.189,"15min":0.183},"max":{"1min":2.2120000000000002,"5min":2.4089999999999998,"15min":6.5960000000000001},"last":0.316}]},{"osd":2,"last update":"Wed Oct 11 11:14:30 2023","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.35499999999999998,"15min":0.47099999999999997},"min":{"1min":0.20300000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":1.2490000000000001,"5min":2.665,"15min":8.4730000000000008},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.375,"15min":0.442},"min":{"1min":0.23200000000000001,"5min":0.20499999999999999,"15min":0.19600000000000001},"max":{"1min":1.28,"5min":2.6230000000000002,"15min":7.0129999999999999},"last":0.33600000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738815,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":826440,"kb_used_data":6392,"kb_used_omap":88,"kb_used_meta":819943,"kb_avail":93545400,"statfs":{"total":96636764160,"available":95790489600,"internally_reserved":0,"allocated":6545408,"data_stored":3697286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":90523,"internal_metadata":839622245},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 11:14:33 2023","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.42799999999999999,"15min":0.41099999999999998},"min":{"1min":0.151,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":1.2809999999999999,"5min":3.5739999999999998,"15min":8.2479999999999993},"last":0.223},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.48299999999999998,"15min":0.49199999999999999},"min":{"1min":0.20699999999999999,"5min":0.182,"15min":0.17999999999999999},"max":{"1min":1.323,"5min":3.6080000000000001,"15min":8.2870000000000008},"last":0.28399999999999997}]},{"osd":2,"last update":"Wed Oct 11 11:19:41 2023","interfaces":[{"interface":"back","average":{"1min":0.314,"5min":0.35399999999999998,"15min":0.35899999999999999},"min":{"1min":0.186,"5min":0.14699999999999999,"15min":0.14699999999999999},"max":{"1min":0.45500000000000002,"5min":7.4829999999999997,"15min":10.617000000000001},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.39400000000000002,"15min":0.35099999999999998},"min":{"1min":0.21199999999999999,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.49099999999999999,"5min":7.5350000000000001,"15min":7.5350000000000001},"last":0.36699999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-11T11:20:33.027 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-10-11T11:20:33.347 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T11:20:33.349 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2023-10-11T11:20:33.361 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":1607,"stamp":"2023-10-11T11:20:32.666112+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":11,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":107,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3096284,"kb_used_data":418996,"kb_used_omap":292,"kb_used_meta":2676955,"kb_avail":280019236,"statfs":{"total":289910292480,"available":286739697664,"internally_reserved":0,"allocated":429051904,"data_stored":420453125,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299441,"internal_metadata":2741202511},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003313"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":517,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-10-11T11:19:02.772828+0000","last_change":"2023-10-11T11:15:31.661747+0000","last_active":"2023-10-11T11:19:02.772828+0000","last_peered":"2023-10-11T11:19:02.772828+0000","last_clean":"2023-10-11T11:15:27.638215+0000","last_became_active":"2023-10-11T11:15:31.661747+0000","last_became_peered":"2023-10-11T11:15:31.661747+0000","last_unstale":"2023-10-11T11:19:02.772828+0000","last_undegraded":"2023-10-11T11:19:02.772828+0000","last_fullsized":"2023-10-11T11:15:31.649336+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:15:25.047039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":29,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-10-11T11:19:02.774213+0000","last_change":"2023-10-11T11:15:31.671363+0000","last_active":"2023-10-11T11:19:02.774213+0000","last_peered":"2023-10-11T11:19:02.774213+0000","last_clean":"2023-10-11T10:43:40.751300+0000","last_became_active":"2023-10-11T11:15:31.671363+0000","last_became_peered":"2023-10-11T11:15:31.671363+0000","last_unstale":"2023-10-11T11:19:02.774213+0000","last_undegraded":"2023-10-11T11:19:02.774213+0000","last_fullsized":"2023-10-11T11:15:31.652001+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:01:53.884964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":512,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-10-11T11:19:03.429640+0000","last_change":"2023-10-11T10:43:41.772781+0000","last_active":"2023-10-11T11:19:03.429640+0000","last_peered":"2023-10-11T11:19:03.429640+0000","last_clean":"2023-10-11T11:19:03.429640+0000","last_became_active":"2023-10-11T10:43:41.772443+0000","last_became_peered":"2023-10-11T10:43:41.772443+0000","last_unstale":"2023-10-11T11:19:03.429640+0000","last_undegraded":"2023-10-11T11:19:03.429640+0000","last_fullsized":"2023-10-11T11:19:03.429640+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T14:15:30.547284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":513,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-10-11T11:19:02.772211+0000","last_change":"2023-10-11T10:43:41.772615+0000","last_active":"2023-10-11T11:19:02.772211+0000","last_peered":"2023-10-11T11:19:02.772211+0000","last_clean":"2023-10-11T11:19:02.772211+0000","last_became_active":"2023-10-11T10:43:41.772290+0000","last_became_peered":"2023-10-11T10:43:41.772290+0000","last_unstale":"2023-10-11T11:19:02.772211+0000","last_undegraded":"2023-10-11T11:19:02.772211+0000","last_fullsized":"2023-10-11T11:19:02.772211+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:26:06.909963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":648,"reported_epoch":500,"state":"active+undersized+degraded","last_fresh":"2023-10-11T11:19:02.769176+0000","last_change":"2023-10-11T11:15:31.654022+0000","last_active":"2023-10-11T11:19:02.769176+0000","last_peered":"2023-10-11T11:19:02.769176+0000","last_clean":"2023-10-11T11:15:27.636884+0000","last_became_active":"2023-10-11T11:15:31.654022+0000","last_became_peered":"2023-10-11T11:15:31.654022+0000","last_unstale":"2023-10-11T11:19:02.769176+0000","last_undegraded":"2023-10-11T11:15:31.646946+0000","last_fullsized":"2023-10-11T11:15:31.646751+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:36.739637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:36.739637+0000","last_clean_scrub_stamp":"2023-10-11T10:43:36.739637+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T11:33:33.696447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":520,"reported_epoch":500,"state":"active+clean+remapped","last_fresh":"2023-10-11T11:19:03.429429+0000","last_change":"2023-10-11T11:15:33.660200+0000","last_active":"2023-10-11T11:19:03.429429+0000","last_peered":"2023-10-11T11:19:03.429429+0000","last_clean":"2023-10-11T11:19:03.429429+0000","last_became_active":"2023-10-11T11:15:33.659619+0000","last_became_peered":"2023-10-11T11:15:33.659619+0000","last_unstale":"2023-10-11T11:19:03.429429+0000","last_undegraded":"2023-10-11T11:19:03.429429+0000","last_fullsized":"2023-10-11T11:19:03.429429+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T17:21:16.866392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":517,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-10-11T11:19:02.775128+0000","last_change":"2023-10-11T11:15:31.670813+0000","last_active":"2023-10-11T11:19:02.775128+0000","last_peered":"2023-10-11T11:19:02.775128+0000","last_clean":"2023-10-11T11:15:27.638219+0000","last_became_active":"2023-10-11T11:15:31.670813+0000","last_became_peered":"2023-10-11T11:15:31.670813+0000","last_unstale":"2023-10-11T11:19:02.775128+0000","last_undegraded":"2023-10-11T11:19:02.775128+0000","last_fullsized":"2023-10-11T11:15:31.658785+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:41:50.872346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":517,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-10-11T11:19:02.772818+0000","last_change":"2023-10-11T11:15:31.666378+0000","last_active":"2023-10-11T11:19:02.772818+0000","last_peered":"2023-10-11T11:19:02.772818+0000","last_clean":"2023-10-11T11:15:27.638303+0000","last_became_active":"2023-10-11T11:15:31.666378+0000","last_became_peered":"2023-10-11T11:15:31.666378+0000","last_unstale":"2023-10-11T11:19:02.772818+0000","last_undegraded":"2023-10-11T11:19:02.772818+0000","last_fullsized":"2023-10-11T11:15:31.649186+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T15:35:19.331209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":516,"reported_epoch":499,"state":"active+undersized","last_fresh":"2023-10-11T11:19:00.636960+0000","last_change":"2023-10-11T11:15:31.679465+0000","last_active":"2023-10-11T11:19:00.636960+0000","last_peered":"2023-10-11T11:19:00.636960+0000","last_clean":"2023-10-11T11:15:27.638749+0000","last_became_active":"2023-10-11T11:15:31.679465+0000","last_became_peered":"2023-10-11T11:15:31.679465+0000","last_unstale":"2023-10-11T11:19:00.636960+0000","last_undegraded":"2023-10-11T11:19:00.636960+0000","last_fullsized":"2023-10-11T11:15:31.659135+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T16:03:42.656692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":11,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738814,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":785348,"kb_used_data":5812,"kb_used_omap":96,"kb_used_meta":779423,"kb_avail":93586492,"statfs":{"total":96636764160,"available":95832567808,"internally_reserved":0,"allocated":5951488,"data_stored":3106918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":98609,"internal_metadata":798129871},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 11:14:37 2023","interfaces":[{"interface":"back","average":{"1min":0.314,"5min":0.316,"15min":0.443},"min":{"1min":0.24399999999999999,"5min":0.14799999999999999,"15min":0.14599999999999999},"max":{"1min":0.48499999999999999,"5min":1.712,"15min":9.6400000000000006},"last":0.31},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.35499999999999998,"15min":0.41299999999999998},"min":{"1min":0.20899999999999999,"5min":0.17199999999999999,"15min":0.16400000000000001},"max":{"1min":3.8180000000000001,"5min":3.8180000000000001,"15min":6.3200000000000003},"last":0.26500000000000001}]},{"osd":1,"last update":"Wed Oct 11 11:19:40 2023","interfaces":[{"interface":"back","average":{"1min":0.29599999999999999,"5min":0.28499999999999998,"15min":0.32300000000000001},"min":{"1min":0.17699999999999999,"5min":0.154,"15min":0.154},"max":{"1min":0.40600000000000003,"5min":0.41999999999999998,"15min":2.9540000000000002},"last":0.22800000000000001},{"interface":"front","average":{"1min":0.317,"5min":0.31,"15min":0.36199999999999999},"min":{"1min":0.20899999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.45000000000000001,"5min":0.49099999999999999,"15min":3.2719999999999998},"last":0.27300000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738756,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1484496,"kb_used_data":406792,"kb_used_omap":107,"kb_used_meta":1077588,"kb_avail":92887344,"statfs":{"total":96636764160,"available":95116640256,"internally_reserved":0,"allocated":416555008,"data_stored":413648921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110309,"internal_metadata":1103450395},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Oct 11 11:14:30 2023","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.35399999999999998,"15min":0.40600000000000003},"min":{"1min":0.18099999999999999,"5min":0.155,"15min":0.14899999999999999},"max":{"1min":1.4319999999999999,"5min":2.3460000000000001,"15min":6.593},"last":0.26200000000000001},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.39100000000000001,"15min":0.48199999999999998},"min":{"1min":0.189,"5min":0.189,"15min":0.183},"max":{"1min":2.2120000000000002,"5min":2.4089999999999998,"15min":6.5960000000000001},"last":0.316}]},{"osd":2,"last update":"Wed Oct 11 11:14:30 2023","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.35499999999999998,"15min":0.47099999999999997},"min":{"1min":0.20300000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":1.2490000000000001,"5min":2.665,"15min":8.4730000000000008},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.375,"15min":0.442},"min":{"1min":0.23200000000000001,"5min":0.20499999999999999,"15min":0.19600000000000001},"max":{"1min":1.28,"5min":2.6230000000000002,"15min":7.0129999999999999},"last":0.33600000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738815,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":826440,"kb_used_data":6392,"kb_used_omap":88,"kb_used_meta":819943,"kb_avail":93545400,"statfs":{"total":96636764160,"available":95790489600,"internally_reserved":0,"allocated":6545408,"data_stored":3697286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":90523,"internal_metadata":839622245},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 11:14:33 2023","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.42799999999999999,"15min":0.41099999999999998},"min":{"1min":0.151,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":1.2809999999999999,"5min":3.5739999999999998,"15min":8.2479999999999993},"last":0.223},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.48299999999999998,"15min":0.49199999999999999},"min":{"1min":0.20699999999999999,"5min":0.182,"15min":0.17999999999999999},"max":{"1min":1.323,"5min":3.6080000000000001,"15min":8.2870000000000008},"last":0.28399999999999997}]},{"osd":2,"last update":"Wed Oct 11 11:19:41 2023","interfaces":[{"interface":"back","average":{"1min":0.314,"5min":0.35399999999999998,"15min":0.35899999999999999},"min":{"1min":0.186,"5min":0.14699999999999999,"15min":0.14699999999999999},"max":{"1min":0.45500000000000002,"5min":7.4829999999999997,"15min":10.617000000000001},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.39400000000000002,"15min":0.35099999999999998},"min":{"1min":0.21199999999999999,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.49099999999999999,"5min":7.5350000000000001,"15min":7.5350000000000001},"last":0.36699999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-11T11:20:33.362 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-10-11T11:20:33.362 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 517, 'reported_epoch': 500, 'state': 'active+undersized', 'last_fresh': '2023-10-11T11:19:02.772828+0000', 'last_change': '2023-10-11T11:15:31.661747+0000', 'last_active': '2023-10-11T11:19:02.772828+0000', 'last_peered': '2023-10-11T11:19:02.772828+0000', 'last_clean': '2023-10-11T11:15:27.638215+0000', 'last_became_active': '2023-10-11T11:15:31.661747+0000', 'last_became_peered': '2023-10-11T11:15:31.661747+0000', 'last_unstale': '2023-10-11T11:19:02.772828+0000', 'last_undegraded': '2023-10-11T11:19:02.772828+0000', 'last_fullsized': '2023-10-11T11:15:31.649336+0000', 'mapping_epoch': 480, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_clean_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-12T22:15:25.047039+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-10-11T11:20:33.362 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-10-11T11:20:33.363 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 29, 'reported_epoch': 500, 'state': 'active+undersized', 'last_fresh': '2023-10-11T11:19:02.774213+0000', 'last_change': '2023-10-11T11:15:31.671363+0000', 'last_active': '2023-10-11T11:19:02.774213+0000', 'last_peered': '2023-10-11T11:19:02.774213+0000', 'last_clean': '2023-10-11T10:43:40.751300+0000', 'last_became_active': '2023-10-11T11:15:31.671363+0000', 'last_became_peered': '2023-10-11T11:15:31.671363+0000', 'last_unstale': '2023-10-11T11:19:02.774213+0000', 'last_undegraded': '2023-10-11T11:19:02.774213+0000', 'last_fullsized': '2023-10-11T11:15:31.652001+0000', 'mapping_epoch': 480, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_clean_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-12T21:01:53.884964+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-10-11T11:20:33.363 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-10-11T11:20:33.363 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "11'97", 'reported_seq': 648, 'reported_epoch': 500, 'state': 'active+undersized+degraded', 'last_fresh': '2023-10-11T11:19:02.769176+0000', 'last_change': '2023-10-11T11:15:31.654022+0000', 'last_active': '2023-10-11T11:19:02.769176+0000', 'last_peered': '2023-10-11T11:19:02.769176+0000', 'last_clean': '2023-10-11T11:15:27.636884+0000', 'last_became_active': '2023-10-11T11:15:31.654022+0000', 'last_became_peered': '2023-10-11T11:15:31.654022+0000', 'last_unstale': '2023-10-11T11:19:02.769176+0000', 'last_undegraded': '2023-10-11T11:15:31.646946+0000', 'last_fullsized': '2023-10-11T11:15:31.646751+0000', 'mapping_epoch': 480, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-11T10:43:36.739637+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-11T10:43:36.739637+0000', 'last_clean_scrub_stamp': '2023-10-11T10:43:36.739637+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-12T11:33:33.696447+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-10-11T11:20:33.363 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.0 is not active+clean 2023-10-11T11:20:33.364 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 520, 'reported_epoch': 500, 'state': 'active+clean+remapped', 'last_fresh': '2023-10-11T11:19:03.429429+0000', 'last_change': '2023-10-11T11:15:33.660200+0000', 'last_active': '2023-10-11T11:19:03.429429+0000', 'last_peered': '2023-10-11T11:19:03.429429+0000', 'last_clean': '2023-10-11T11:19:03.429429+0000', 'last_became_active': '2023-10-11T11:15:33.659619+0000', 'last_became_peered': '2023-10-11T11:15:33.659619+0000', 'last_unstale': '2023-10-11T11:19:03.429429+0000', 'last_undegraded': '2023-10-11T11:19:03.429429+0000', 'last_fullsized': '2023-10-11T11:19:03.429429+0000', 'mapping_epoch': 482, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 483, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_clean_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-12T17:21:16.866392+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2, 1], 'avail_no_missing': ['2', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-10-11T11:20:33.364 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-10-11T11:20:33.364 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 517, 'reported_epoch': 500, 'state': 'active+undersized', 'last_fresh': '2023-10-11T11:19:02.775128+0000', 'last_change': '2023-10-11T11:15:31.670813+0000', 'last_active': '2023-10-11T11:19:02.775128+0000', 'last_peered': '2023-10-11T11:19:02.775128+0000', 'last_clean': '2023-10-11T11:15:27.638219+0000', 'last_became_active': '2023-10-11T11:15:31.670813+0000', 'last_became_peered': '2023-10-11T11:15:31.670813+0000', 'last_unstale': '2023-10-11T11:19:02.775128+0000', 'last_undegraded': '2023-10-11T11:19:02.775128+0000', 'last_fullsized': '2023-10-11T11:15:31.658785+0000', 'mapping_epoch': 480, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_clean_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-12T21:41:50.872346+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-10-11T11:20:33.364 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-10-11T11:20:33.365 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 517, 'reported_epoch': 500, 'state': 'active+undersized', 'last_fresh': '2023-10-11T11:19:02.772818+0000', 'last_change': '2023-10-11T11:15:31.666378+0000', 'last_active': '2023-10-11T11:19:02.772818+0000', 'last_peered': '2023-10-11T11:19:02.772818+0000', 'last_clean': '2023-10-11T11:15:27.638303+0000', 'last_became_active': '2023-10-11T11:15:31.666378+0000', 'last_became_peered': '2023-10-11T11:15:31.666378+0000', 'last_unstale': '2023-10-11T11:19:02.772818+0000', 'last_undegraded': '2023-10-11T11:19:02.772818+0000', 'last_fullsized': '2023-10-11T11:15:31.649186+0000', 'mapping_epoch': 480, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_clean_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-12T15:35:19.331209+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-10-11T11:20:33.365 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-10-11T11:20:33.365 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 516, 'reported_epoch': 499, 'state': 'active+undersized', 'last_fresh': '2023-10-11T11:19:00.636960+0000', 'last_change': '2023-10-11T11:15:31.679465+0000', 'last_active': '2023-10-11T11:19:00.636960+0000', 'last_peered': '2023-10-11T11:19:00.636960+0000', 'last_clean': '2023-10-11T11:15:27.638749+0000', 'last_became_active': '2023-10-11T11:15:31.679465+0000', 'last_became_peered': '2023-10-11T11:15:31.679465+0000', 'last_unstale': '2023-10-11T11:19:00.636960+0000', 'last_undegraded': '2023-10-11T11:19:00.636960+0000', 'last_fullsized': '2023-10-11T11:15:31.659135+0000', 'mapping_epoch': 480, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'last_clean_scrub_stamp': '2023-10-11T10:43:40.751300+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-12T16:03:42.656692+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-10-11T11:20:33.366 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-10-11T11:20:33.761 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T11:20:33.761 INFO:teuthology.orchestra.run.smithi111.stdout:{"fsid":"022ce6f2-9a5b-45ad-8b77-a1332d92f228","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2228,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":500,"num_osds":3,"num_up_osds":2,"osd_up_since":1697022930,"num_in_osds":3,"osd_in_since":1697021005,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":2},{"state_name":"active+undersized+degraded","count":1},{"state_name":"active+clean+remapped","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3170594816,"bytes_avail":286739697664,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":36,"modified":"2023-10-11T11:19:04.644408+0000","services":{}},"progress_events":{}} 2023-10-11T11:20:33.775 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-10-11T11:20:33.775 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-10-11T11:20:34.090 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T11:20:34.092 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2023-10-11T11:20:34.104 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":1607,"stamp":"2023-10-11T11:20:32.666112+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":11,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":107,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3096284,"kb_used_data":418996,"kb_used_omap":292,"kb_used_meta":2676955,"kb_avail":280019236,"statfs":{"total":289910292480,"available":286739697664,"internally_reserved":0,"allocated":429051904,"data_stored":420453125,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299441,"internal_metadata":2741202511},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003313"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":517,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-10-11T11:19:02.772828+0000","last_change":"2023-10-11T11:15:31.661747+0000","last_active":"2023-10-11T11:19:02.772828+0000","last_peered":"2023-10-11T11:19:02.772828+0000","last_clean":"2023-10-11T11:15:27.638215+0000","last_became_active":"2023-10-11T11:15:31.661747+0000","last_became_peered":"2023-10-11T11:15:31.661747+0000","last_unstale":"2023-10-11T11:19:02.772828+0000","last_undegraded":"2023-10-11T11:19:02.772828+0000","last_fullsized":"2023-10-11T11:15:31.649336+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:15:25.047039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":29,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-10-11T11:19:02.774213+0000","last_change":"2023-10-11T11:15:31.671363+0000","last_active":"2023-10-11T11:19:02.774213+0000","last_peered":"2023-10-11T11:19:02.774213+0000","last_clean":"2023-10-11T10:43:40.751300+0000","last_became_active":"2023-10-11T11:15:31.671363+0000","last_became_peered":"2023-10-11T11:15:31.671363+0000","last_unstale":"2023-10-11T11:19:02.774213+0000","last_undegraded":"2023-10-11T11:19:02.774213+0000","last_fullsized":"2023-10-11T11:15:31.652001+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:01:53.884964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":512,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-10-11T11:19:03.429640+0000","last_change":"2023-10-11T10:43:41.772781+0000","last_active":"2023-10-11T11:19:03.429640+0000","last_peered":"2023-10-11T11:19:03.429640+0000","last_clean":"2023-10-11T11:19:03.429640+0000","last_became_active":"2023-10-11T10:43:41.772443+0000","last_became_peered":"2023-10-11T10:43:41.772443+0000","last_unstale":"2023-10-11T11:19:03.429640+0000","last_undegraded":"2023-10-11T11:19:03.429640+0000","last_fullsized":"2023-10-11T11:19:03.429640+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T14:15:30.547284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":513,"reported_epoch":500,"state":"active+clean","last_fresh":"2023-10-11T11:19:02.772211+0000","last_change":"2023-10-11T10:43:41.772615+0000","last_active":"2023-10-11T11:19:02.772211+0000","last_peered":"2023-10-11T11:19:02.772211+0000","last_clean":"2023-10-11T11:19:02.772211+0000","last_became_active":"2023-10-11T10:43:41.772290+0000","last_became_peered":"2023-10-11T10:43:41.772290+0000","last_unstale":"2023-10-11T11:19:02.772211+0000","last_undegraded":"2023-10-11T11:19:02.772211+0000","last_fullsized":"2023-10-11T11:19:02.772211+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:26:06.909963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":648,"reported_epoch":500,"state":"active+undersized+degraded","last_fresh":"2023-10-11T11:19:02.769176+0000","last_change":"2023-10-11T11:15:31.654022+0000","last_active":"2023-10-11T11:19:02.769176+0000","last_peered":"2023-10-11T11:19:02.769176+0000","last_clean":"2023-10-11T11:15:27.636884+0000","last_became_active":"2023-10-11T11:15:31.654022+0000","last_became_peered":"2023-10-11T11:15:31.654022+0000","last_unstale":"2023-10-11T11:19:02.769176+0000","last_undegraded":"2023-10-11T11:15:31.646946+0000","last_fullsized":"2023-10-11T11:15:31.646751+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:36.739637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:36.739637+0000","last_clean_scrub_stamp":"2023-10-11T10:43:36.739637+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T11:33:33.696447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":520,"reported_epoch":500,"state":"active+clean+remapped","last_fresh":"2023-10-11T11:19:03.429429+0000","last_change":"2023-10-11T11:15:33.660200+0000","last_active":"2023-10-11T11:19:03.429429+0000","last_peered":"2023-10-11T11:19:03.429429+0000","last_clean":"2023-10-11T11:19:03.429429+0000","last_became_active":"2023-10-11T11:15:33.659619+0000","last_became_peered":"2023-10-11T11:15:33.659619+0000","last_unstale":"2023-10-11T11:19:03.429429+0000","last_undegraded":"2023-10-11T11:19:03.429429+0000","last_fullsized":"2023-10-11T11:19:03.429429+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T17:21:16.866392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":517,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-10-11T11:19:02.775128+0000","last_change":"2023-10-11T11:15:31.670813+0000","last_active":"2023-10-11T11:19:02.775128+0000","last_peered":"2023-10-11T11:19:02.775128+0000","last_clean":"2023-10-11T11:15:27.638219+0000","last_became_active":"2023-10-11T11:15:31.670813+0000","last_became_peered":"2023-10-11T11:15:31.670813+0000","last_unstale":"2023-10-11T11:19:02.775128+0000","last_undegraded":"2023-10-11T11:19:02.775128+0000","last_fullsized":"2023-10-11T11:15:31.658785+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:41:50.872346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":517,"reported_epoch":500,"state":"active+undersized","last_fresh":"2023-10-11T11:19:02.772818+0000","last_change":"2023-10-11T11:15:31.666378+0000","last_active":"2023-10-11T11:19:02.772818+0000","last_peered":"2023-10-11T11:19:02.772818+0000","last_clean":"2023-10-11T11:15:27.638303+0000","last_became_active":"2023-10-11T11:15:31.666378+0000","last_became_peered":"2023-10-11T11:15:31.666378+0000","last_unstale":"2023-10-11T11:19:02.772818+0000","last_undegraded":"2023-10-11T11:19:02.772818+0000","last_fullsized":"2023-10-11T11:15:31.649186+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T15:35:19.331209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":516,"reported_epoch":499,"state":"active+undersized","last_fresh":"2023-10-11T11:19:00.636960+0000","last_change":"2023-10-11T11:15:31.679465+0000","last_active":"2023-10-11T11:19:00.636960+0000","last_peered":"2023-10-11T11:19:00.636960+0000","last_clean":"2023-10-11T11:15:27.638749+0000","last_became_active":"2023-10-11T11:15:31.679465+0000","last_became_peered":"2023-10-11T11:15:31.679465+0000","last_unstale":"2023-10-11T11:19:00.636960+0000","last_undegraded":"2023-10-11T11:19:00.636960+0000","last_fullsized":"2023-10-11T11:15:31.659135+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:40.751300+0000","last_clean_scrub_stamp":"2023-10-11T10:43:40.751300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T16:03:42.656692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":11,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738814,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":785348,"kb_used_data":5812,"kb_used_omap":96,"kb_used_meta":779423,"kb_avail":93586492,"statfs":{"total":96636764160,"available":95832567808,"internally_reserved":0,"allocated":5951488,"data_stored":3106918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":98609,"internal_metadata":798129871},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 11:14:37 2023","interfaces":[{"interface":"back","average":{"1min":0.314,"5min":0.316,"15min":0.443},"min":{"1min":0.24399999999999999,"5min":0.14799999999999999,"15min":0.14599999999999999},"max":{"1min":0.48499999999999999,"5min":1.712,"15min":9.6400000000000006},"last":0.31},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.35499999999999998,"15min":0.41299999999999998},"min":{"1min":0.20899999999999999,"5min":0.17199999999999999,"15min":0.16400000000000001},"max":{"1min":3.8180000000000001,"5min":3.8180000000000001,"15min":6.3200000000000003},"last":0.26500000000000001}]},{"osd":1,"last update":"Wed Oct 11 11:19:40 2023","interfaces":[{"interface":"back","average":{"1min":0.29599999999999999,"5min":0.28499999999999998,"15min":0.32300000000000001},"min":{"1min":0.17699999999999999,"5min":0.154,"15min":0.154},"max":{"1min":0.40600000000000003,"5min":0.41999999999999998,"15min":2.9540000000000002},"last":0.22800000000000001},{"interface":"front","average":{"1min":0.317,"5min":0.31,"15min":0.36199999999999999},"min":{"1min":0.20899999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.45000000000000001,"5min":0.49099999999999999,"15min":3.2719999999999998},"last":0.27300000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738756,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1484496,"kb_used_data":406792,"kb_used_omap":107,"kb_used_meta":1077588,"kb_avail":92887344,"statfs":{"total":96636764160,"available":95116640256,"internally_reserved":0,"allocated":416555008,"data_stored":413648921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110309,"internal_metadata":1103450395},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Oct 11 11:14:30 2023","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.35399999999999998,"15min":0.40600000000000003},"min":{"1min":0.18099999999999999,"5min":0.155,"15min":0.14899999999999999},"max":{"1min":1.4319999999999999,"5min":2.3460000000000001,"15min":6.593},"last":0.26200000000000001},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.39100000000000001,"15min":0.48199999999999998},"min":{"1min":0.189,"5min":0.189,"15min":0.183},"max":{"1min":2.2120000000000002,"5min":2.4089999999999998,"15min":6.5960000000000001},"last":0.316}]},{"osd":2,"last update":"Wed Oct 11 11:14:30 2023","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.35499999999999998,"15min":0.47099999999999997},"min":{"1min":0.20300000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":1.2490000000000001,"5min":2.665,"15min":8.4730000000000008},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.375,"15min":0.442},"min":{"1min":0.23200000000000001,"5min":0.20499999999999999,"15min":0.19600000000000001},"max":{"1min":1.28,"5min":2.6230000000000002,"15min":7.0129999999999999},"last":0.33600000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738815,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":826440,"kb_used_data":6392,"kb_used_omap":88,"kb_used_meta":819943,"kb_avail":93545400,"statfs":{"total":96636764160,"available":95790489600,"internally_reserved":0,"allocated":6545408,"data_stored":3697286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":90523,"internal_metadata":839622245},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 11:14:33 2023","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.42799999999999999,"15min":0.41099999999999998},"min":{"1min":0.151,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":1.2809999999999999,"5min":3.5739999999999998,"15min":8.2479999999999993},"last":0.223},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.48299999999999998,"15min":0.49199999999999999},"min":{"1min":0.20699999999999999,"5min":0.182,"15min":0.17999999999999999},"max":{"1min":1.323,"5min":3.6080000000000001,"15min":8.2870000000000008},"last":0.28399999999999997}]},{"osd":2,"last update":"Wed Oct 11 11:19:41 2023","interfaces":[{"interface":"back","average":{"1min":0.314,"5min":0.35399999999999998,"15min":0.35899999999999999},"min":{"1min":0.186,"5min":0.14699999999999999,"15min":0.14699999999999999},"max":{"1min":0.45500000000000002,"5min":7.4829999999999997,"15min":10.617000000000001},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.39400000000000002,"15min":0.35099999999999998},"min":{"1min":0.21199999999999999,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.49099999999999999,"5min":7.5350000000000001,"15min":7.5350000000000001},"last":0.36699999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-11T11:20:34.773 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~303s 2023-10-11T11:20:34.773 INFO:tasks.daemonwatchdog.daemon_watchdog:BARK! unmounting mounts and killing all daemons 2023-10-11T11:20:35.177 INFO:tasks.ceph.cluster2.osd.1:Sent signal 15 2023-10-11T11:20:35.178 INFO:tasks.ceph.cluster2.osd.2:Sent signal 15 2023-10-11T11:20:35.178 INFO:tasks.ceph.cluster2.mon.a:Sent signal 15 2023-10-11T11:20:35.179 INFO:tasks.ceph.cluster2.mgr.x:Sent signal 15 2023-10-11T11:20:35.180 INFO:tasks.ceph.cluster2.osd.1.smithi111.stderr:2023-10-11T11:20:35.177+0000 7f9636a0b700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster cluster2 -i 1 (PID: 100704) UID: 0 2023-10-11T11:20:35.180 INFO:tasks.ceph.cluster2.osd.1.smithi111.stderr:2023-10-11T11:20:35.177+0000 7f9636a0b700 -1 osd.1 500 *** Got signal Terminated *** 2023-10-11T11:20:35.180 INFO:tasks.ceph.cluster2.osd.1.smithi111.stderr:2023-10-11T11:20:35.177+0000 7f9636a0b700 -1 osd.1 500 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-10-11T11:20:35.181 INFO:tasks.ceph.cluster2.osd.2.smithi111.stderr:2023-10-11T11:20:35.178+0000 7fa36a092700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster cluster2 -i 2 (PID: 100709) UID: 0 2023-10-11T11:20:35.181 INFO:tasks.ceph.cluster2.osd.2.smithi111.stderr:2023-10-11T11:20:35.178+0000 7fa36a092700 -1 osd.2 500 *** Got signal Terminated *** 2023-10-11T11:20:35.181 INFO:tasks.ceph.cluster2.osd.2.smithi111.stderr:2023-10-11T11:20:35.178+0000 7fa36a092700 -1 osd.2 500 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-10-11T11:20:35.182 INFO:tasks.ceph.cluster2.mon.a.smithi111.stderr:2023-10-11T11:20:35.178+0000 7f9f8486b700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mon -f --cluster cluster2 -i a (PID: 100364) UID: 0 2023-10-11T11:20:35.182 INFO:tasks.ceph.cluster2.mon.a.smithi111.stderr:2023-10-11T11:20:35.178+0000 7f9f8486b700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2023-10-11T11:20:35.182 INFO:tasks.ceph.cluster2.mgr.x.smithi050.stderr:2023-10-11T11:20:35.180+0000 7fa018e48700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mgr -f --cluster cluster2 -i x (PID: 105286) UID: 0 2023-10-11T11:20:35.183 INFO:tasks.ceph.cluster2.mgr.x.smithi050.stderr:2023-10-11T11:20:35.180+0000 7fa018e48700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2023-10-11T11:20:37.106 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-10-11T11:22:37.144 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-10-11T11:22:37.146 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_idryomov_ceph_3d099a3859cc211cf8dd2acdaf3194ba0e96ea9c/qa/tasks/ceph.py", line 1950, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_idryomov_ceph_3d099a3859cc211cf8dd2acdaf3194ba0e96ea9c/qa/tasks/ceph_manager.py", line 2743, in wait_for_clean while not self.is_clean(): File "/home/teuthworker/src/github.com_idryomov_ceph_3d099a3859cc211cf8dd2acdaf3194ba0e96ea9c/qa/tasks/ceph_manager.py", line 2679, in is_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_idryomov_ceph_3d099a3859cc211cf8dd2acdaf3194ba0e96ea9c/qa/tasks/ceph_manager.py", line 2340, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_idryomov_ceph_3d099a3859cc211cf8dd2acdaf3194ba0e96ea9c/qa/tasks/ceph_manager.py", line 1595, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_idryomov_ceph_3d099a3859cc211cf8dd2acdaf3194ba0e96ea9c/qa/tasks/ceph_manager.py", line 1586, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi111 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json' 2023-10-11T11:22:37.146 INFO:teuthology.misc:Shutting down mds daemons... 2023-10-11T11:22:37.147 INFO:teuthology.misc:Shutting down osd daemons... 2023-10-11T11:22:37.147 DEBUG:tasks.ceph.cluster2.osd.0:waiting for process to exit 2023-10-11T11:22:37.148 INFO:teuthology.orchestra.run:waiting for 300 2023-10-11T11:22:37.148 DEBUG:teuthology.orchestra.run:got remote process result: 5 2023-10-11T11:22:37.148 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/github.com_idryomov_ceph_3d099a3859cc211cf8dd2acdaf3194ba0e96ea9c/qa/tasks/ceph.py", line 1469, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_idryomov_ceph_3d099a3859cc211cf8dd2acdaf3194ba0e96ea9c/qa/tasks/ceph.py", line 1950, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_idryomov_ceph_3d099a3859cc211cf8dd2acdaf3194ba0e96ea9c/qa/tasks/ceph_manager.py", line 2743, in wait_for_clean while not self.is_clean(): File "/home/teuthworker/src/github.com_idryomov_ceph_3d099a3859cc211cf8dd2acdaf3194ba0e96ea9c/qa/tasks/ceph_manager.py", line 2679, in is_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_idryomov_ceph_3d099a3859cc211cf8dd2acdaf3194ba0e96ea9c/qa/tasks/ceph_manager.py", line 2340, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_idryomov_ceph_3d099a3859cc211cf8dd2acdaf3194ba0e96ea9c/qa/tasks/ceph_manager.py", line 1595, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_idryomov_ceph_3d099a3859cc211cf8dd2acdaf3194ba0e96ea9c/qa/tasks/ceph_manager.py", line 1586, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi111 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi111 with status 5: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster2 -i 0' 2023-10-11T11:22:37.149 INFO:tasks.ceph.cluster2.osd.0:Stopped 2023-10-11T11:22:37.150 DEBUG:tasks.ceph.cluster2.osd.1:waiting for process to exit 2023-10-11T11:22:37.150 INFO:teuthology.orchestra.run:waiting for 300 2023-10-11T11:22:37.151 INFO:tasks.ceph.cluster2.osd.1:Stopped 2023-10-11T11:22:37.151 DEBUG:tasks.ceph.cluster2.osd.2:waiting for process to exit 2023-10-11T11:22:37.151 INFO:teuthology.orchestra.run:waiting for 300 2023-10-11T11:22:37.152 INFO:tasks.ceph.cluster2.osd.2:Stopped 2023-10-11T11:22:37.152 INFO:teuthology.misc:Shutting down mgr daemons... 2023-10-11T11:22:37.153 DEBUG:tasks.ceph.cluster2.mgr.x:waiting for process to exit 2023-10-11T11:22:37.153 INFO:teuthology.orchestra.run:waiting for 300 2023-10-11T11:22:37.154 INFO:tasks.ceph.cluster2.mgr.x:Stopped 2023-10-11T11:22:37.154 INFO:teuthology.misc:Shutting down mon daemons... 2023-10-11T11:22:37.155 DEBUG:tasks.ceph.cluster2.mon.a:waiting for process to exit 2023-10-11T11:22:37.155 INFO:teuthology.orchestra.run:waiting for 300 2023-10-11T11:22:37.156 INFO:tasks.ceph.cluster2.mon.a:Stopped 2023-10-11T11:22:37.156 INFO:tasks.ceph:Checking cluster log for badness... 2023-10-11T11:22:37.157 DEBUG:teuthology.orchestra.run.smithi111:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/cluster2.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | head -n 1 2023-10-11T11:22:37.191 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster2-0 on ubuntu@smithi111.front.sepia.ceph.com 2023-10-11T11:22:37.192 DEBUG:teuthology.orchestra.run.smithi111:> sync && sudo umount -f /var/lib/ceph/osd/cluster2-0 2023-10-11T11:22:37.486 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster2-1 on ubuntu@smithi111.front.sepia.ceph.com 2023-10-11T11:22:37.487 DEBUG:teuthology.orchestra.run.smithi111:> sync && sudo umount -f /var/lib/ceph/osd/cluster2-1 2023-10-11T11:22:37.723 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster2-2 on ubuntu@smithi111.front.sepia.ceph.com 2023-10-11T11:22:37.724 DEBUG:teuthology.orchestra.run.smithi111:> sync && sudo umount -f /var/lib/ceph/osd/cluster2-2 2023-10-11T11:22:37.902 INFO:tasks.ceph:Archiving mon data... 2023-10-11T11:22:37.903 DEBUG:teuthology.misc:Transferring archived files from smithi111:/var/lib/ceph/mon/cluster2-a to /home/teuthworker/archive/dis-2023-10-11_10:22:06-rbd-wip-dis-testing-distro-default-smithi/7421102/data/cluster2.mon.a.tgz 2023-10-11T11:22:37.904 DEBUG:teuthology.orchestra.run.smithi111:> mktemp 2023-10-11T11:22:37.921 INFO:teuthology.orchestra.run.smithi111.stdout:/tmp/tmp.Sm1ivTyTUZ 2023-10-11T11:22:37.921 DEBUG:teuthology.orchestra.run.smithi111:> sudo tar cz -f - -C /var/lib/ceph/mon/cluster2-a -- . > /tmp/tmp.Sm1ivTyTUZ 2023-10-11T11:22:38.086 DEBUG:teuthology.orchestra.run.smithi111:> sudo chmod 0666 /tmp/tmp.Sm1ivTyTUZ 2023-10-11T11:22:38.171 DEBUG:teuthology.orchestra.remote:smithi111:/tmp/tmp.Sm1ivTyTUZ is 460KB 2023-10-11T11:22:38.270 DEBUG:teuthology.orchestra.run.smithi111:> rm -fr /tmp/tmp.Sm1ivTyTUZ 2023-10-11T11:22:38.288 INFO:tasks.ceph:Cleaning ceph cluster... 2023-10-11T11:22:38.288 DEBUG:teuthology.orchestra.run.smithi050:> sudo rm -rf -- /etc/ceph/cluster2.conf /etc/ceph/cluster2.keyring /home/ubuntu/cephtest/cluster2.data /home/ubuntu/cephtest/cluster2.monmap /home/ubuntu/cephtest/../*.pid 2023-10-11T11:22:38.291 DEBUG:teuthology.orchestra.run.smithi111:> sudo rm -rf -- /etc/ceph/cluster2.conf /etc/ceph/cluster2.keyring /home/ubuntu/cephtest/cluster2.data /home/ubuntu/cephtest/cluster2.monmap /home/ubuntu/cephtest/../*.pid 2023-10-11T11:22:38.353 ERROR:teuthology.run_tasks:Manager failed: ceph Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_idryomov_ceph_3d099a3859cc211cf8dd2acdaf3194ba0e96ea9c/qa/tasks/ceph.py", line 1950, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_idryomov_ceph_3d099a3859cc211cf8dd2acdaf3194ba0e96ea9c/qa/tasks/ceph_manager.py", line 2743, in wait_for_clean while not self.is_clean(): File "/home/teuthworker/src/github.com_idryomov_ceph_3d099a3859cc211cf8dd2acdaf3194ba0e96ea9c/qa/tasks/ceph_manager.py", line 2679, in is_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_idryomov_ceph_3d099a3859cc211cf8dd2acdaf3194ba0e96ea9c/qa/tasks/ceph_manager.py", line 2340, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_idryomov_ceph_3d099a3859cc211cf8dd2acdaf3194ba0e96ea9c/qa/tasks/ceph_manager.py", line 1595, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_idryomov_ceph_3d099a3859cc211cf8dd2acdaf3194ba0e96ea9c/qa/tasks/ceph_manager.py", line 1586, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi111 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json' 2023-10-11T11:22:38.354 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-10-11T11:22:38.369 INFO:tasks.ceph.ceph_manager.cluster1:waiting for clean 2023-10-11T11:22:38.369 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2023-10-11T11:22:38.711 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-11T11:22:38.713 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2023-10-11T11:22:38.724 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":1654,"stamp":"2023-10-11T11:22:37.103860+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":456988,"kb_used_data":12408,"kb_used_omap":4,"kb_used_meta":444539,"kb_avail":282658532,"statfs":{"total":289910292480,"available":289442336768,"internally_reserved":0,"allocated":12705792,"data_stored":7714902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4782,"internal_metadata":455208274},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002342"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":481,"reported_epoch":467,"state":"active+clean","last_fresh":"2023-10-11T11:19:01.850102+0000","last_change":"2023-10-11T10:43:03.433134+0000","last_active":"2023-10-11T11:19:01.850102+0000","last_peered":"2023-10-11T11:19:01.850102+0000","last_clean":"2023-10-11T11:19:01.850102+0000","last_became_active":"2023-10-11T10:43:03.432855+0000","last_became_peered":"2023-10-11T10:43:03.432855+0000","last_unstale":"2023-10-11T11:19:01.850102+0000","last_undegraded":"2023-10-11T11:19:01.850102+0000","last_fullsized":"2023-10-11T11:19:01.850102+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:02.419376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:02.419376+0000","last_clean_scrub_stamp":"2023-10-11T10:43:02.419376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T20:03:03.803743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":480,"reported_epoch":467,"state":"active+clean","last_fresh":"2023-10-11T11:19:03.990776+0000","last_change":"2023-10-11T10:43:03.434716+0000","last_active":"2023-10-11T11:19:03.990776+0000","last_peered":"2023-10-11T11:19:03.990776+0000","last_clean":"2023-10-11T11:19:03.990776+0000","last_became_active":"2023-10-11T10:43:03.434392+0000","last_became_peered":"2023-10-11T10:43:03.434392+0000","last_unstale":"2023-10-11T11:19:03.990776+0000","last_undegraded":"2023-10-11T11:19:03.990776+0000","last_fullsized":"2023-10-11T11:19:03.990776+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:02.419376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:02.419376+0000","last_clean_scrub_stamp":"2023-10-11T10:43:02.419376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T13:50:37.330835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":482,"reported_epoch":467,"state":"active+clean","last_fresh":"2023-10-11T11:20:31.592081+0000","last_change":"2023-10-11T10:43:03.434605+0000","last_active":"2023-10-11T11:20:31.592081+0000","last_peered":"2023-10-11T11:20:31.592081+0000","last_clean":"2023-10-11T11:20:31.592081+0000","last_became_active":"2023-10-11T10:43:03.434104+0000","last_became_peered":"2023-10-11T10:43:03.434104+0000","last_unstale":"2023-10-11T11:20:31.592081+0000","last_undegraded":"2023-10-11T11:20:31.592081+0000","last_fullsized":"2023-10-11T11:20:31.592081+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:02.419376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:02.419376+0000","last_clean_scrub_stamp":"2023-10-11T10:43:02.419376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T13:35:08.276972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":481,"reported_epoch":467,"state":"active+clean","last_fresh":"2023-10-11T11:19:01.849434+0000","last_change":"2023-10-11T10:43:03.433261+0000","last_active":"2023-10-11T11:19:01.849434+0000","last_peered":"2023-10-11T11:19:01.849434+0000","last_clean":"2023-10-11T11:19:01.849434+0000","last_became_active":"2023-10-11T10:43:03.432640+0000","last_became_peered":"2023-10-11T10:43:03.432640+0000","last_unstale":"2023-10-11T11:19:01.849434+0000","last_undegraded":"2023-10-11T11:19:01.849434+0000","last_fullsized":"2023-10-11T11:19:01.849434+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:02.419376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:02.419376+0000","last_clean_scrub_stamp":"2023-10-11T10:43:02.419376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T18:19:21.672854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":612,"reported_epoch":467,"state":"active+clean","last_fresh":"2023-10-11T11:19:06.764378+0000","last_change":"2023-10-11T10:42:57.665559+0000","last_active":"2023-10-11T11:19:06.764378+0000","last_peered":"2023-10-11T11:19:06.764378+0000","last_clean":"2023-10-11T11:19:06.764378+0000","last_became_active":"2023-10-11T10:42:57.664210+0000","last_became_peered":"2023-10-11T10:42:57.664210+0000","last_unstale":"2023-10-11T11:19:06.764378+0000","last_undegraded":"2023-10-11T11:19:06.764378+0000","last_fullsized":"2023-10-11T11:19:06.764378+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:42:56.657464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:42:56.657464+0000","last_clean_scrub_stamp":"2023-10-11T10:42:56.657464+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T15:54:11.939340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":482,"reported_epoch":467,"state":"active+clean","last_fresh":"2023-10-11T11:19:16.582647+0000","last_change":"2023-10-11T10:43:03.434801+0000","last_active":"2023-10-11T11:19:16.582647+0000","last_peered":"2023-10-11T11:19:16.582647+0000","last_clean":"2023-10-11T11:19:16.582647+0000","last_became_active":"2023-10-11T10:43:03.434471+0000","last_became_peered":"2023-10-11T10:43:03.434471+0000","last_unstale":"2023-10-11T11:19:16.582647+0000","last_undegraded":"2023-10-11T11:19:16.582647+0000","last_fullsized":"2023-10-11T11:19:16.582647+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:02.419376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:02.419376+0000","last_clean_scrub_stamp":"2023-10-11T10:43:02.419376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:09:57.094086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":481,"reported_epoch":467,"state":"active+clean","last_fresh":"2023-10-11T11:19:01.851645+0000","last_change":"2023-10-11T10:43:03.435783+0000","last_active":"2023-10-11T11:19:01.851645+0000","last_peered":"2023-10-11T11:19:01.851645+0000","last_clean":"2023-10-11T11:19:01.851645+0000","last_became_active":"2023-10-11T10:43:03.435469+0000","last_became_peered":"2023-10-11T10:43:03.435469+0000","last_unstale":"2023-10-11T11:19:01.851645+0000","last_undegraded":"2023-10-11T11:19:01.851645+0000","last_fullsized":"2023-10-11T11:19:01.851645+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:02.419376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:02.419376+0000","last_clean_scrub_stamp":"2023-10-11T10:43:02.419376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T17:08:03.285598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":481,"reported_epoch":467,"state":"active+clean","last_fresh":"2023-10-11T11:19:51.769442+0000","last_change":"2023-10-11T10:43:03.432737+0000","last_active":"2023-10-11T11:19:51.769442+0000","last_peered":"2023-10-11T11:19:51.769442+0000","last_clean":"2023-10-11T11:19:51.769442+0000","last_became_active":"2023-10-11T10:43:03.432238+0000","last_became_peered":"2023-10-11T10:43:03.432238+0000","last_unstale":"2023-10-11T11:19:51.769442+0000","last_undegraded":"2023-10-11T11:19:51.769442+0000","last_fullsized":"2023-10-11T11:19:51.769442+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:02.419376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:02.419376+0000","last_clean_scrub_stamp":"2023-10-11T10:43:02.419376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T20:21:46.997470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":481,"reported_epoch":467,"state":"active+clean","last_fresh":"2023-10-11T11:19:56.769946+0000","last_change":"2023-10-11T10:43:03.434569+0000","last_active":"2023-10-11T11:19:56.769946+0000","last_peered":"2023-10-11T11:19:56.769946+0000","last_clean":"2023-10-11T11:19:56.769946+0000","last_became_active":"2023-10-11T10:43:03.434147+0000","last_became_peered":"2023-10-11T10:43:03.434147+0000","last_unstale":"2023-10-11T11:19:56.769946+0000","last_undegraded":"2023-10-11T11:19:56.769946+0000","last_fullsized":"2023-10-11T11:19:56.769946+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:02.419376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:02.419376+0000","last_clean_scrub_stamp":"2023-10-11T10:43:02.419376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:12:12.693291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738850,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":151472,"kb_used_data":3748,"kb_used_omap":1,"kb_used_meta":147710,"kb_avail":94220368,"statfs":{"total":96636764160,"available":96481656832,"internally_reserved":0,"allocated":3837952,"data_stored":2177799,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":151255494},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 11:22:08 2023","interfaces":[{"interface":"back","average":{"1min":0.36099999999999999,"5min":0.34499999999999997,"15min":0.32900000000000001},"min":{"1min":0.218,"5min":0.21199999999999999,"15min":0.17499999999999999},"max":{"1min":0.53000000000000003,"5min":0.61599999999999999,"15min":1.143},"last":0.28599999999999998},{"interface":"front","average":{"1min":0.41299999999999998,"5min":0.40699999999999997,"15min":0.39000000000000001},"min":{"1min":0.32100000000000001,"5min":0.25900000000000001,"15min":0.255},"max":{"1min":0.54700000000000004,"5min":0.70799999999999996,"15min":0.78900000000000003},"last":0.38700000000000001}]},{"osd":1,"last update":"Wed Oct 11 11:22:08 2023","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.36799999999999999,"15min":0.35199999999999998},"min":{"1min":0.28299999999999997,"5min":0.25800000000000001,"15min":0.222},"max":{"1min":0.53600000000000003,"5min":0.57599999999999996,"15min":0.68700000000000006},"last":0.31},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.36599999999999999,"15min":0.34799999999999998},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.20999999999999999},"max":{"1min":0.51000000000000001,"5min":0.54100000000000004,"15min":1.2090000000000001},"last":0.34699999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738848,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":157620,"kb_used_data":4328,"kb_used_omap":1,"kb_used_meta":153278,"kb_avail":94214220,"statfs":{"total":96636764160,"available":96475361280,"internally_reserved":0,"allocated":4431872,"data_stored":2768167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":156957126},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Oct 11 11:22:14 2023","interfaces":[{"interface":"back","average":{"1min":0.32500000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"min":{"1min":0.23599999999999999,"5min":0.23300000000000001,"15min":0.183},"max":{"1min":0.45500000000000002,"5min":0.5,"15min":3.3340000000000001},"last":0.32600000000000001},{"interface":"front","average":{"1min":0.40200000000000002,"5min":0.40400000000000003,"15min":0.40500000000000003},"min":{"1min":0.311,"5min":0.28499999999999998,"15min":0.23400000000000001},"max":{"1min":0.53300000000000003,"5min":0.54200000000000004,"15min":3.3839999999999999},"last":0.41699999999999998}]},{"osd":2,"last update":"Wed Oct 11 11:22:14 2023","interfaces":[{"interface":"back","average":{"1min":0.35299999999999998,"5min":0.35199999999999998,"15min":0.34599999999999997},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.20200000000000001},"max":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":1.895},"last":0.33900000000000002},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.36799999999999999,"15min":0.36699999999999999},"min":{"1min":0.27000000000000002,"5min":0.26500000000000001,"15min":0.20699999999999999},"max":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":2.3580000000000001},"last":0.39700000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738847,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":147896,"kb_used_data":4332,"kb_used_omap":1,"kb_used_meta":143550,"kb_avail":94223944,"statfs":{"total":96636764160,"available":96485318656,"internally_reserved":0,"allocated":4435968,"data_stored":2768936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":146995654},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 11:22:14 2023","interfaces":[{"interface":"back","average":{"1min":0.376,"5min":0.35199999999999998,"15min":0.32900000000000001},"min":{"1min":0.23899999999999999,"5min":0.20399999999999999,"15min":0.19800000000000001},"max":{"1min":0.5,"5min":0.55600000000000005,"15min":0.55600000000000005},"last":0.29899999999999999},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.35899999999999999,"15min":0.33900000000000002},"min":{"1min":0.27200000000000002,"5min":0.24399999999999999,"15min":0.193},"max":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"last":0.32100000000000001}]},{"osd":2,"last update":"Wed Oct 11 11:22:14 2023","interfaces":[{"interface":"back","average":{"1min":0.39700000000000002,"5min":0.378,"15min":0.35499999999999998},"min":{"1min":0.27900000000000003,"5min":0.22,"15min":0.21199999999999999},"max":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.55300000000000005},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.39900000000000002,"5min":0.377,"15min":0.35799999999999998},"min":{"1min":0.28599999999999998,"5min":0.251,"15min":0.22500000000000001},"max":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.69899999999999995},"last":0.38500000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-11T11:22:38.726 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2023-10-11T11:22:39.058 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-11T11:22:39.060 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2023-10-11T11:22:39.072 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":1654,"stamp":"2023-10-11T11:22:37.103860+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":456988,"kb_used_data":12408,"kb_used_omap":4,"kb_used_meta":444539,"kb_avail":282658532,"statfs":{"total":289910292480,"available":289442336768,"internally_reserved":0,"allocated":12705792,"data_stored":7714902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4782,"internal_metadata":455208274},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002342"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":481,"reported_epoch":467,"state":"active+clean","last_fresh":"2023-10-11T11:19:01.850102+0000","last_change":"2023-10-11T10:43:03.433134+0000","last_active":"2023-10-11T11:19:01.850102+0000","last_peered":"2023-10-11T11:19:01.850102+0000","last_clean":"2023-10-11T11:19:01.850102+0000","last_became_active":"2023-10-11T10:43:03.432855+0000","last_became_peered":"2023-10-11T10:43:03.432855+0000","last_unstale":"2023-10-11T11:19:01.850102+0000","last_undegraded":"2023-10-11T11:19:01.850102+0000","last_fullsized":"2023-10-11T11:19:01.850102+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:02.419376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:02.419376+0000","last_clean_scrub_stamp":"2023-10-11T10:43:02.419376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T20:03:03.803743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":480,"reported_epoch":467,"state":"active+clean","last_fresh":"2023-10-11T11:19:03.990776+0000","last_change":"2023-10-11T10:43:03.434716+0000","last_active":"2023-10-11T11:19:03.990776+0000","last_peered":"2023-10-11T11:19:03.990776+0000","last_clean":"2023-10-11T11:19:03.990776+0000","last_became_active":"2023-10-11T10:43:03.434392+0000","last_became_peered":"2023-10-11T10:43:03.434392+0000","last_unstale":"2023-10-11T11:19:03.990776+0000","last_undegraded":"2023-10-11T11:19:03.990776+0000","last_fullsized":"2023-10-11T11:19:03.990776+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:02.419376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:02.419376+0000","last_clean_scrub_stamp":"2023-10-11T10:43:02.419376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T13:50:37.330835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":482,"reported_epoch":467,"state":"active+clean","last_fresh":"2023-10-11T11:20:31.592081+0000","last_change":"2023-10-11T10:43:03.434605+0000","last_active":"2023-10-11T11:20:31.592081+0000","last_peered":"2023-10-11T11:20:31.592081+0000","last_clean":"2023-10-11T11:20:31.592081+0000","last_became_active":"2023-10-11T10:43:03.434104+0000","last_became_peered":"2023-10-11T10:43:03.434104+0000","last_unstale":"2023-10-11T11:20:31.592081+0000","last_undegraded":"2023-10-11T11:20:31.592081+0000","last_fullsized":"2023-10-11T11:20:31.592081+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:02.419376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:02.419376+0000","last_clean_scrub_stamp":"2023-10-11T10:43:02.419376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T13:35:08.276972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":481,"reported_epoch":467,"state":"active+clean","last_fresh":"2023-10-11T11:19:01.849434+0000","last_change":"2023-10-11T10:43:03.433261+0000","last_active":"2023-10-11T11:19:01.849434+0000","last_peered":"2023-10-11T11:19:01.849434+0000","last_clean":"2023-10-11T11:19:01.849434+0000","last_became_active":"2023-10-11T10:43:03.432640+0000","last_became_peered":"2023-10-11T10:43:03.432640+0000","last_unstale":"2023-10-11T11:19:01.849434+0000","last_undegraded":"2023-10-11T11:19:01.849434+0000","last_fullsized":"2023-10-11T11:19:01.849434+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:02.419376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:02.419376+0000","last_clean_scrub_stamp":"2023-10-11T10:43:02.419376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T18:19:21.672854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":612,"reported_epoch":467,"state":"active+clean","last_fresh":"2023-10-11T11:19:06.764378+0000","last_change":"2023-10-11T10:42:57.665559+0000","last_active":"2023-10-11T11:19:06.764378+0000","last_peered":"2023-10-11T11:19:06.764378+0000","last_clean":"2023-10-11T11:19:06.764378+0000","last_became_active":"2023-10-11T10:42:57.664210+0000","last_became_peered":"2023-10-11T10:42:57.664210+0000","last_unstale":"2023-10-11T11:19:06.764378+0000","last_undegraded":"2023-10-11T11:19:06.764378+0000","last_fullsized":"2023-10-11T11:19:06.764378+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:42:56.657464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:42:56.657464+0000","last_clean_scrub_stamp":"2023-10-11T10:42:56.657464+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T15:54:11.939340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":482,"reported_epoch":467,"state":"active+clean","last_fresh":"2023-10-11T11:19:16.582647+0000","last_change":"2023-10-11T10:43:03.434801+0000","last_active":"2023-10-11T11:19:16.582647+0000","last_peered":"2023-10-11T11:19:16.582647+0000","last_clean":"2023-10-11T11:19:16.582647+0000","last_became_active":"2023-10-11T10:43:03.434471+0000","last_became_peered":"2023-10-11T10:43:03.434471+0000","last_unstale":"2023-10-11T11:19:16.582647+0000","last_undegraded":"2023-10-11T11:19:16.582647+0000","last_fullsized":"2023-10-11T11:19:16.582647+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:02.419376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:02.419376+0000","last_clean_scrub_stamp":"2023-10-11T10:43:02.419376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:09:57.094086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":481,"reported_epoch":467,"state":"active+clean","last_fresh":"2023-10-11T11:19:01.851645+0000","last_change":"2023-10-11T10:43:03.435783+0000","last_active":"2023-10-11T11:19:01.851645+0000","last_peered":"2023-10-11T11:19:01.851645+0000","last_clean":"2023-10-11T11:19:01.851645+0000","last_became_active":"2023-10-11T10:43:03.435469+0000","last_became_peered":"2023-10-11T10:43:03.435469+0000","last_unstale":"2023-10-11T11:19:01.851645+0000","last_undegraded":"2023-10-11T11:19:01.851645+0000","last_fullsized":"2023-10-11T11:19:01.851645+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:02.419376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:02.419376+0000","last_clean_scrub_stamp":"2023-10-11T10:43:02.419376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T17:08:03.285598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":481,"reported_epoch":467,"state":"active+clean","last_fresh":"2023-10-11T11:19:51.769442+0000","last_change":"2023-10-11T10:43:03.432737+0000","last_active":"2023-10-11T11:19:51.769442+0000","last_peered":"2023-10-11T11:19:51.769442+0000","last_clean":"2023-10-11T11:19:51.769442+0000","last_became_active":"2023-10-11T10:43:03.432238+0000","last_became_peered":"2023-10-11T10:43:03.432238+0000","last_unstale":"2023-10-11T11:19:51.769442+0000","last_undegraded":"2023-10-11T11:19:51.769442+0000","last_fullsized":"2023-10-11T11:19:51.769442+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:02.419376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:02.419376+0000","last_clean_scrub_stamp":"2023-10-11T10:43:02.419376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T20:21:46.997470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":481,"reported_epoch":467,"state":"active+clean","last_fresh":"2023-10-11T11:19:56.769946+0000","last_change":"2023-10-11T10:43:03.434569+0000","last_active":"2023-10-11T11:19:56.769946+0000","last_peered":"2023-10-11T11:19:56.769946+0000","last_clean":"2023-10-11T11:19:56.769946+0000","last_became_active":"2023-10-11T10:43:03.434147+0000","last_became_peered":"2023-10-11T10:43:03.434147+0000","last_unstale":"2023-10-11T11:19:56.769946+0000","last_undegraded":"2023-10-11T11:19:56.769946+0000","last_fullsized":"2023-10-11T11:19:56.769946+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:02.419376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:02.419376+0000","last_clean_scrub_stamp":"2023-10-11T10:43:02.419376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:12:12.693291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738850,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":151472,"kb_used_data":3748,"kb_used_omap":1,"kb_used_meta":147710,"kb_avail":94220368,"statfs":{"total":96636764160,"available":96481656832,"internally_reserved":0,"allocated":3837952,"data_stored":2177799,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":151255494},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 11:22:08 2023","interfaces":[{"interface":"back","average":{"1min":0.36099999999999999,"5min":0.34499999999999997,"15min":0.32900000000000001},"min":{"1min":0.218,"5min":0.21199999999999999,"15min":0.17499999999999999},"max":{"1min":0.53000000000000003,"5min":0.61599999999999999,"15min":1.143},"last":0.28599999999999998},{"interface":"front","average":{"1min":0.41299999999999998,"5min":0.40699999999999997,"15min":0.39000000000000001},"min":{"1min":0.32100000000000001,"5min":0.25900000000000001,"15min":0.255},"max":{"1min":0.54700000000000004,"5min":0.70799999999999996,"15min":0.78900000000000003},"last":0.38700000000000001}]},{"osd":1,"last update":"Wed Oct 11 11:22:08 2023","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.36799999999999999,"15min":0.35199999999999998},"min":{"1min":0.28299999999999997,"5min":0.25800000000000001,"15min":0.222},"max":{"1min":0.53600000000000003,"5min":0.57599999999999996,"15min":0.68700000000000006},"last":0.31},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.36599999999999999,"15min":0.34799999999999998},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.20999999999999999},"max":{"1min":0.51000000000000001,"5min":0.54100000000000004,"15min":1.2090000000000001},"last":0.34699999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738848,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":157620,"kb_used_data":4328,"kb_used_omap":1,"kb_used_meta":153278,"kb_avail":94214220,"statfs":{"total":96636764160,"available":96475361280,"internally_reserved":0,"allocated":4431872,"data_stored":2768167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":156957126},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Oct 11 11:22:14 2023","interfaces":[{"interface":"back","average":{"1min":0.32500000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"min":{"1min":0.23599999999999999,"5min":0.23300000000000001,"15min":0.183},"max":{"1min":0.45500000000000002,"5min":0.5,"15min":3.3340000000000001},"last":0.32600000000000001},{"interface":"front","average":{"1min":0.40200000000000002,"5min":0.40400000000000003,"15min":0.40500000000000003},"min":{"1min":0.311,"5min":0.28499999999999998,"15min":0.23400000000000001},"max":{"1min":0.53300000000000003,"5min":0.54200000000000004,"15min":3.3839999999999999},"last":0.41699999999999998}]},{"osd":2,"last update":"Wed Oct 11 11:22:14 2023","interfaces":[{"interface":"back","average":{"1min":0.35299999999999998,"5min":0.35199999999999998,"15min":0.34599999999999997},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.20200000000000001},"max":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":1.895},"last":0.33900000000000002},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.36799999999999999,"15min":0.36699999999999999},"min":{"1min":0.27000000000000002,"5min":0.26500000000000001,"15min":0.20699999999999999},"max":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":2.3580000000000001},"last":0.39700000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738847,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":147896,"kb_used_data":4332,"kb_used_omap":1,"kb_used_meta":143550,"kb_avail":94223944,"statfs":{"total":96636764160,"available":96485318656,"internally_reserved":0,"allocated":4435968,"data_stored":2768936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":146995654},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 11:22:14 2023","interfaces":[{"interface":"back","average":{"1min":0.376,"5min":0.35199999999999998,"15min":0.32900000000000001},"min":{"1min":0.23899999999999999,"5min":0.20399999999999999,"15min":0.19800000000000001},"max":{"1min":0.5,"5min":0.55600000000000005,"15min":0.55600000000000005},"last":0.29899999999999999},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.35899999999999999,"15min":0.33900000000000002},"min":{"1min":0.27200000000000002,"5min":0.24399999999999999,"15min":0.193},"max":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"last":0.32100000000000001}]},{"osd":2,"last update":"Wed Oct 11 11:22:14 2023","interfaces":[{"interface":"back","average":{"1min":0.39700000000000002,"5min":0.378,"15min":0.35499999999999998},"min":{"1min":0.27900000000000003,"5min":0.22,"15min":0.21199999999999999},"max":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.55300000000000005},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.39900000000000002,"5min":0.377,"15min":0.35799999999999998},"min":{"1min":0.28599999999999998,"5min":0.251,"15min":0.22500000000000001},"max":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.69899999999999995},"last":0.38500000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-11T11:22:39.073 INFO:tasks.ceph.ceph_manager.cluster1:clean! 2023-10-11T11:22:39.074 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2023-10-11T11:22:39.406 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-11T11:22:39.408 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2023-10-11T11:22:39.419 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":1655,"stamp":"2023-10-11T11:22:39.104155+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":456988,"kb_used_data":12408,"kb_used_omap":4,"kb_used_meta":444539,"kb_avail":282658532,"statfs":{"total":289910292480,"available":289442336768,"internally_reserved":0,"allocated":12705792,"data_stored":7714902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4782,"internal_metadata":455208274},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002269"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":481,"reported_epoch":467,"state":"active+clean","last_fresh":"2023-10-11T11:19:01.850102+0000","last_change":"2023-10-11T10:43:03.433134+0000","last_active":"2023-10-11T11:19:01.850102+0000","last_peered":"2023-10-11T11:19:01.850102+0000","last_clean":"2023-10-11T11:19:01.850102+0000","last_became_active":"2023-10-11T10:43:03.432855+0000","last_became_peered":"2023-10-11T10:43:03.432855+0000","last_unstale":"2023-10-11T11:19:01.850102+0000","last_undegraded":"2023-10-11T11:19:01.850102+0000","last_fullsized":"2023-10-11T11:19:01.850102+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:02.419376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:02.419376+0000","last_clean_scrub_stamp":"2023-10-11T10:43:02.419376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T20:03:03.803743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":480,"reported_epoch":467,"state":"active+clean","last_fresh":"2023-10-11T11:19:03.990776+0000","last_change":"2023-10-11T10:43:03.434716+0000","last_active":"2023-10-11T11:19:03.990776+0000","last_peered":"2023-10-11T11:19:03.990776+0000","last_clean":"2023-10-11T11:19:03.990776+0000","last_became_active":"2023-10-11T10:43:03.434392+0000","last_became_peered":"2023-10-11T10:43:03.434392+0000","last_unstale":"2023-10-11T11:19:03.990776+0000","last_undegraded":"2023-10-11T11:19:03.990776+0000","last_fullsized":"2023-10-11T11:19:03.990776+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:02.419376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:02.419376+0000","last_clean_scrub_stamp":"2023-10-11T10:43:02.419376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T13:50:37.330835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":482,"reported_epoch":467,"state":"active+clean","last_fresh":"2023-10-11T11:20:31.592081+0000","last_change":"2023-10-11T10:43:03.434605+0000","last_active":"2023-10-11T11:20:31.592081+0000","last_peered":"2023-10-11T11:20:31.592081+0000","last_clean":"2023-10-11T11:20:31.592081+0000","last_became_active":"2023-10-11T10:43:03.434104+0000","last_became_peered":"2023-10-11T10:43:03.434104+0000","last_unstale":"2023-10-11T11:20:31.592081+0000","last_undegraded":"2023-10-11T11:20:31.592081+0000","last_fullsized":"2023-10-11T11:20:31.592081+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:02.419376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:02.419376+0000","last_clean_scrub_stamp":"2023-10-11T10:43:02.419376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T13:35:08.276972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":481,"reported_epoch":467,"state":"active+clean","last_fresh":"2023-10-11T11:19:01.849434+0000","last_change":"2023-10-11T10:43:03.433261+0000","last_active":"2023-10-11T11:19:01.849434+0000","last_peered":"2023-10-11T11:19:01.849434+0000","last_clean":"2023-10-11T11:19:01.849434+0000","last_became_active":"2023-10-11T10:43:03.432640+0000","last_became_peered":"2023-10-11T10:43:03.432640+0000","last_unstale":"2023-10-11T11:19:01.849434+0000","last_undegraded":"2023-10-11T11:19:01.849434+0000","last_fullsized":"2023-10-11T11:19:01.849434+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:02.419376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:02.419376+0000","last_clean_scrub_stamp":"2023-10-11T10:43:02.419376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T18:19:21.672854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":612,"reported_epoch":467,"state":"active+clean","last_fresh":"2023-10-11T11:19:06.764378+0000","last_change":"2023-10-11T10:42:57.665559+0000","last_active":"2023-10-11T11:19:06.764378+0000","last_peered":"2023-10-11T11:19:06.764378+0000","last_clean":"2023-10-11T11:19:06.764378+0000","last_became_active":"2023-10-11T10:42:57.664210+0000","last_became_peered":"2023-10-11T10:42:57.664210+0000","last_unstale":"2023-10-11T11:19:06.764378+0000","last_undegraded":"2023-10-11T11:19:06.764378+0000","last_fullsized":"2023-10-11T11:19:06.764378+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:42:56.657464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:42:56.657464+0000","last_clean_scrub_stamp":"2023-10-11T10:42:56.657464+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T15:54:11.939340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":482,"reported_epoch":467,"state":"active+clean","last_fresh":"2023-10-11T11:19:16.582647+0000","last_change":"2023-10-11T10:43:03.434801+0000","last_active":"2023-10-11T11:19:16.582647+0000","last_peered":"2023-10-11T11:19:16.582647+0000","last_clean":"2023-10-11T11:19:16.582647+0000","last_became_active":"2023-10-11T10:43:03.434471+0000","last_became_peered":"2023-10-11T10:43:03.434471+0000","last_unstale":"2023-10-11T11:19:16.582647+0000","last_undegraded":"2023-10-11T11:19:16.582647+0000","last_fullsized":"2023-10-11T11:19:16.582647+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:02.419376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:02.419376+0000","last_clean_scrub_stamp":"2023-10-11T10:43:02.419376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:09:57.094086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":481,"reported_epoch":467,"state":"active+clean","last_fresh":"2023-10-11T11:19:01.851645+0000","last_change":"2023-10-11T10:43:03.435783+0000","last_active":"2023-10-11T11:19:01.851645+0000","last_peered":"2023-10-11T11:19:01.851645+0000","last_clean":"2023-10-11T11:19:01.851645+0000","last_became_active":"2023-10-11T10:43:03.435469+0000","last_became_peered":"2023-10-11T10:43:03.435469+0000","last_unstale":"2023-10-11T11:19:01.851645+0000","last_undegraded":"2023-10-11T11:19:01.851645+0000","last_fullsized":"2023-10-11T11:19:01.851645+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:02.419376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:02.419376+0000","last_clean_scrub_stamp":"2023-10-11T10:43:02.419376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T17:08:03.285598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":481,"reported_epoch":467,"state":"active+clean","last_fresh":"2023-10-11T11:19:51.769442+0000","last_change":"2023-10-11T10:43:03.432737+0000","last_active":"2023-10-11T11:19:51.769442+0000","last_peered":"2023-10-11T11:19:51.769442+0000","last_clean":"2023-10-11T11:19:51.769442+0000","last_became_active":"2023-10-11T10:43:03.432238+0000","last_became_peered":"2023-10-11T10:43:03.432238+0000","last_unstale":"2023-10-11T11:19:51.769442+0000","last_undegraded":"2023-10-11T11:19:51.769442+0000","last_fullsized":"2023-10-11T11:19:51.769442+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:02.419376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:02.419376+0000","last_clean_scrub_stamp":"2023-10-11T10:43:02.419376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T20:21:46.997470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":481,"reported_epoch":467,"state":"active+clean","last_fresh":"2023-10-11T11:19:56.769946+0000","last_change":"2023-10-11T10:43:03.434569+0000","last_active":"2023-10-11T11:19:56.769946+0000","last_peered":"2023-10-11T11:19:56.769946+0000","last_clean":"2023-10-11T11:19:56.769946+0000","last_became_active":"2023-10-11T10:43:03.434147+0000","last_became_peered":"2023-10-11T10:43:03.434147+0000","last_unstale":"2023-10-11T11:19:56.769946+0000","last_undegraded":"2023-10-11T11:19:56.769946+0000","last_fullsized":"2023-10-11T11:19:56.769946+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:02.419376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:02.419376+0000","last_clean_scrub_stamp":"2023-10-11T10:43:02.419376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:12:12.693291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738850,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":151472,"kb_used_data":3748,"kb_used_omap":1,"kb_used_meta":147710,"kb_avail":94220368,"statfs":{"total":96636764160,"available":96481656832,"internally_reserved":0,"allocated":3837952,"data_stored":2177799,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":151255494},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 11:22:08 2023","interfaces":[{"interface":"back","average":{"1min":0.36099999999999999,"5min":0.34499999999999997,"15min":0.32900000000000001},"min":{"1min":0.218,"5min":0.21199999999999999,"15min":0.17499999999999999},"max":{"1min":0.53000000000000003,"5min":0.61599999999999999,"15min":1.143},"last":0.28599999999999998},{"interface":"front","average":{"1min":0.41299999999999998,"5min":0.40699999999999997,"15min":0.39000000000000001},"min":{"1min":0.32100000000000001,"5min":0.25900000000000001,"15min":0.255},"max":{"1min":0.54700000000000004,"5min":0.70799999999999996,"15min":0.78900000000000003},"last":0.38700000000000001}]},{"osd":1,"last update":"Wed Oct 11 11:22:08 2023","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.36799999999999999,"15min":0.35199999999999998},"min":{"1min":0.28299999999999997,"5min":0.25800000000000001,"15min":0.222},"max":{"1min":0.53600000000000003,"5min":0.57599999999999996,"15min":0.68700000000000006},"last":0.31},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.36599999999999999,"15min":0.34799999999999998},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.20999999999999999},"max":{"1min":0.51000000000000001,"5min":0.54100000000000004,"15min":1.2090000000000001},"last":0.34699999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738848,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":157620,"kb_used_data":4328,"kb_used_omap":1,"kb_used_meta":153278,"kb_avail":94214220,"statfs":{"total":96636764160,"available":96475361280,"internally_reserved":0,"allocated":4431872,"data_stored":2768167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":156957126},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Oct 11 11:22:14 2023","interfaces":[{"interface":"back","average":{"1min":0.32500000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"min":{"1min":0.23599999999999999,"5min":0.23300000000000001,"15min":0.183},"max":{"1min":0.45500000000000002,"5min":0.5,"15min":3.3340000000000001},"last":0.32600000000000001},{"interface":"front","average":{"1min":0.40200000000000002,"5min":0.40400000000000003,"15min":0.40500000000000003},"min":{"1min":0.311,"5min":0.28499999999999998,"15min":0.23400000000000001},"max":{"1min":0.53300000000000003,"5min":0.54200000000000004,"15min":3.3839999999999999},"last":0.41699999999999998}]},{"osd":2,"last update":"Wed Oct 11 11:22:14 2023","interfaces":[{"interface":"back","average":{"1min":0.35299999999999998,"5min":0.35199999999999998,"15min":0.34599999999999997},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.20200000000000001},"max":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":1.895},"last":0.33900000000000002},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.36799999999999999,"15min":0.36699999999999999},"min":{"1min":0.27000000000000002,"5min":0.26500000000000001,"15min":0.20699999999999999},"max":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":2.3580000000000001},"last":0.39700000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738847,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":147896,"kb_used_data":4332,"kb_used_omap":1,"kb_used_meta":143550,"kb_avail":94223944,"statfs":{"total":96636764160,"available":96485318656,"internally_reserved":0,"allocated":4435968,"data_stored":2768936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":146995654},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 11:22:14 2023","interfaces":[{"interface":"back","average":{"1min":0.376,"5min":0.35199999999999998,"15min":0.32900000000000001},"min":{"1min":0.23899999999999999,"5min":0.20399999999999999,"15min":0.19800000000000001},"max":{"1min":0.5,"5min":0.55600000000000005,"15min":0.55600000000000005},"last":0.29899999999999999},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.35899999999999999,"15min":0.33900000000000002},"min":{"1min":0.27200000000000002,"5min":0.24399999999999999,"15min":0.193},"max":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"last":0.32100000000000001}]},{"osd":2,"last update":"Wed Oct 11 11:22:14 2023","interfaces":[{"interface":"back","average":{"1min":0.39700000000000002,"5min":0.378,"15min":0.35499999999999998},"min":{"1min":0.27900000000000003,"5min":0.22,"15min":0.21199999999999999},"max":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.55300000000000005},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.39900000000000002,"5min":0.377,"15min":0.35799999999999998},"min":{"1min":0.28599999999999998,"5min":0.251,"15min":0.22500000000000001},"max":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.69899999999999995},"last":0.38500000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-11T11:22:39.421 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2023-10-11T11:22:39.759 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-11T11:22:39.760 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":467,"fsid":"62c62aca-a3d3-4cfb-94c6-84fa1a529031","created":"2023-10-11T10:42:46.445350+0000","modified":"2023-10-11T11:19:01.838877+0000","last_up_change":"2023-10-11T10:42:55.656744+0000","last_in_change":"2023-10-11T10:42:47.449931+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-11T10:42:56.555720+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-11T10:43:01.959424+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4a4611ce-7da6-470a-8649-b63d56727239","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":711452231},{"type":"v1","addr":"172.21.15.50:6817","nonce":711452231}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":711452231},{"type":"v1","addr":"172.21.15.50:6819","nonce":711452231}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6822","nonce":711452231},{"type":"v1","addr":"172.21.15.50:6823","nonce":711452231}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":711452231},{"type":"v1","addr":"172.21.15.50:6821","nonce":711452231}]},"public_addr":"172.21.15.50:6817/711452231","cluster_addr":"172.21.15.50:6819/711452231","heartbeat_back_addr":"172.21.15.50:6823/711452231","heartbeat_front_addr":"172.21.15.50:6821/711452231","state":["exists","up"]},{"osd":1,"uuid":"49a99803-504b-40ed-9ab9-62d95fe81988","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":130890563},{"type":"v1","addr":"172.21.15.50:6801","nonce":130890563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":130890563},{"type":"v1","addr":"172.21.15.50:6803","nonce":130890563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":130890563},{"type":"v1","addr":"172.21.15.50:6807","nonce":130890563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":130890563},{"type":"v1","addr":"172.21.15.50:6805","nonce":130890563}]},"public_addr":"172.21.15.50:6801/130890563","cluster_addr":"172.21.15.50:6803/130890563","heartbeat_back_addr":"172.21.15.50:6807/130890563","heartbeat_front_addr":"172.21.15.50:6805/130890563","state":["exists","up"]},{"osd":2,"uuid":"69abfaca-41cc-4e5b-91c5-98dfa3531012","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":1081260863},{"type":"v1","addr":"172.21.15.50:6809","nonce":1081260863}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":1081260863},{"type":"v1","addr":"172.21.15.50:6811","nonce":1081260863}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":1081260863},{"type":"v1","addr":"172.21.15.50:6815","nonce":1081260863}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":1081260863},{"type":"v1","addr":"172.21.15.50:6813","nonce":1081260863}]},"public_addr":"172.21.15.50:6809/1081260863","cluster_addr":"172.21.15.50:6811/1081260863","heartbeat_back_addr":"172.21.15.50:6815/1081260863","heartbeat_front_addr":"172.21.15.50:6813/1081260863","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T10:42:52.641134+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T10:42:52.631329+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T10:42:52.679138+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":390,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-11T11:22:40.774 INFO:tasks.ceph:Scrubbing cluster1.osd.0 2023-10-11T11:22:40.775 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.0 config set osd_debug_deep_scrub_sleep 0 2023-10-11T11:22:40.940 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-11T11:22:40.941 INFO:teuthology.orchestra.run.smithi050.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-11T11:22:40.941 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-11T11:22:40.952 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd deep-scrub 0 2023-10-11T11:22:41.292 INFO:teuthology.orchestra.run.smithi050.stderr:instructed osd(s) 0 to deep-scrub 2023-10-11T11:22:41.303 INFO:tasks.ceph:Scrubbing cluster1.osd.1 2023-10-11T11:22:41.304 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.1 config set osd_debug_deep_scrub_sleep 0 2023-10-11T11:22:41.454 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-11T11:22:41.454 INFO:teuthology.orchestra.run.smithi050.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-11T11:22:41.455 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-11T11:22:41.466 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd deep-scrub 1 2023-10-11T11:22:41.810 INFO:teuthology.orchestra.run.smithi050.stderr:instructed osd(s) 1 to deep-scrub 2023-10-11T11:22:41.821 INFO:tasks.ceph:Scrubbing cluster1.osd.2 2023-10-11T11:22:41.821 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.2 config set osd_debug_deep_scrub_sleep 0 2023-10-11T11:22:41.971 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-11T11:22:41.972 INFO:teuthology.orchestra.run.smithi050.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-11T11:22:41.972 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-11T11:22:41.983 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd deep-scrub 2 2023-10-11T11:22:42.326 INFO:teuthology.orchestra.run.smithi050.stderr:instructed osd(s) 2 to deep-scrub 2023-10-11T11:22:42.337 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2023-10-11T11:22:42.669 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-11T11:22:42.671 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2023-10-11T11:22:42.683 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":1656,"stamp":"2023-10-11T11:22:41.104413+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":456988,"kb_used_data":12408,"kb_used_omap":4,"kb_used_meta":444539,"kb_avail":282658532,"statfs":{"total":289910292480,"available":289442336768,"internally_reserved":0,"allocated":12705792,"data_stored":7714902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4782,"internal_metadata":455208274},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002113"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":481,"reported_epoch":467,"state":"active+clean","last_fresh":"2023-10-11T11:19:01.850102+0000","last_change":"2023-10-11T10:43:03.433134+0000","last_active":"2023-10-11T11:19:01.850102+0000","last_peered":"2023-10-11T11:19:01.850102+0000","last_clean":"2023-10-11T11:19:01.850102+0000","last_became_active":"2023-10-11T10:43:03.432855+0000","last_became_peered":"2023-10-11T10:43:03.432855+0000","last_unstale":"2023-10-11T11:19:01.850102+0000","last_undegraded":"2023-10-11T11:19:01.850102+0000","last_fullsized":"2023-10-11T11:19:01.850102+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:02.419376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:02.419376+0000","last_clean_scrub_stamp":"2023-10-11T10:43:02.419376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T20:03:03.803743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":480,"reported_epoch":467,"state":"active+clean","last_fresh":"2023-10-11T11:19:03.990776+0000","last_change":"2023-10-11T10:43:03.434716+0000","last_active":"2023-10-11T11:19:03.990776+0000","last_peered":"2023-10-11T11:19:03.990776+0000","last_clean":"2023-10-11T11:19:03.990776+0000","last_became_active":"2023-10-11T10:43:03.434392+0000","last_became_peered":"2023-10-11T10:43:03.434392+0000","last_unstale":"2023-10-11T11:19:03.990776+0000","last_undegraded":"2023-10-11T11:19:03.990776+0000","last_fullsized":"2023-10-11T11:19:03.990776+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:02.419376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:02.419376+0000","last_clean_scrub_stamp":"2023-10-11T10:43:02.419376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T13:50:37.330835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":482,"reported_epoch":467,"state":"active+clean","last_fresh":"2023-10-11T11:20:31.592081+0000","last_change":"2023-10-11T10:43:03.434605+0000","last_active":"2023-10-11T11:20:31.592081+0000","last_peered":"2023-10-11T11:20:31.592081+0000","last_clean":"2023-10-11T11:20:31.592081+0000","last_became_active":"2023-10-11T10:43:03.434104+0000","last_became_peered":"2023-10-11T10:43:03.434104+0000","last_unstale":"2023-10-11T11:20:31.592081+0000","last_undegraded":"2023-10-11T11:20:31.592081+0000","last_fullsized":"2023-10-11T11:20:31.592081+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:02.419376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:02.419376+0000","last_clean_scrub_stamp":"2023-10-11T10:43:02.419376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T13:35:08.276972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":481,"reported_epoch":467,"state":"active+clean","last_fresh":"2023-10-11T11:19:01.849434+0000","last_change":"2023-10-11T10:43:03.433261+0000","last_active":"2023-10-11T11:19:01.849434+0000","last_peered":"2023-10-11T11:19:01.849434+0000","last_clean":"2023-10-11T11:19:01.849434+0000","last_became_active":"2023-10-11T10:43:03.432640+0000","last_became_peered":"2023-10-11T10:43:03.432640+0000","last_unstale":"2023-10-11T11:19:01.849434+0000","last_undegraded":"2023-10-11T11:19:01.849434+0000","last_fullsized":"2023-10-11T11:19:01.849434+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:02.419376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:02.419376+0000","last_clean_scrub_stamp":"2023-10-11T10:43:02.419376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T18:19:21.672854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":612,"reported_epoch":467,"state":"active+clean","last_fresh":"2023-10-11T11:19:06.764378+0000","last_change":"2023-10-11T10:42:57.665559+0000","last_active":"2023-10-11T11:19:06.764378+0000","last_peered":"2023-10-11T11:19:06.764378+0000","last_clean":"2023-10-11T11:19:06.764378+0000","last_became_active":"2023-10-11T10:42:57.664210+0000","last_became_peered":"2023-10-11T10:42:57.664210+0000","last_unstale":"2023-10-11T11:19:06.764378+0000","last_undegraded":"2023-10-11T11:19:06.764378+0000","last_fullsized":"2023-10-11T11:19:06.764378+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:42:56.657464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:42:56.657464+0000","last_clean_scrub_stamp":"2023-10-11T10:42:56.657464+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T15:54:11.939340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":482,"reported_epoch":467,"state":"active+clean","last_fresh":"2023-10-11T11:19:16.582647+0000","last_change":"2023-10-11T10:43:03.434801+0000","last_active":"2023-10-11T11:19:16.582647+0000","last_peered":"2023-10-11T11:19:16.582647+0000","last_clean":"2023-10-11T11:19:16.582647+0000","last_became_active":"2023-10-11T10:43:03.434471+0000","last_became_peered":"2023-10-11T10:43:03.434471+0000","last_unstale":"2023-10-11T11:19:16.582647+0000","last_undegraded":"2023-10-11T11:19:16.582647+0000","last_fullsized":"2023-10-11T11:19:16.582647+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:02.419376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:02.419376+0000","last_clean_scrub_stamp":"2023-10-11T10:43:02.419376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:09:57.094086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":481,"reported_epoch":467,"state":"active+clean","last_fresh":"2023-10-11T11:19:01.851645+0000","last_change":"2023-10-11T10:43:03.435783+0000","last_active":"2023-10-11T11:19:01.851645+0000","last_peered":"2023-10-11T11:19:01.851645+0000","last_clean":"2023-10-11T11:19:01.851645+0000","last_became_active":"2023-10-11T10:43:03.435469+0000","last_became_peered":"2023-10-11T10:43:03.435469+0000","last_unstale":"2023-10-11T11:19:01.851645+0000","last_undegraded":"2023-10-11T11:19:01.851645+0000","last_fullsized":"2023-10-11T11:19:01.851645+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:02.419376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:02.419376+0000","last_clean_scrub_stamp":"2023-10-11T10:43:02.419376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T17:08:03.285598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":481,"reported_epoch":467,"state":"active+clean","last_fresh":"2023-10-11T11:19:51.769442+0000","last_change":"2023-10-11T10:43:03.432737+0000","last_active":"2023-10-11T11:19:51.769442+0000","last_peered":"2023-10-11T11:19:51.769442+0000","last_clean":"2023-10-11T11:19:51.769442+0000","last_became_active":"2023-10-11T10:43:03.432238+0000","last_became_peered":"2023-10-11T10:43:03.432238+0000","last_unstale":"2023-10-11T11:19:51.769442+0000","last_undegraded":"2023-10-11T11:19:51.769442+0000","last_fullsized":"2023-10-11T11:19:51.769442+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:02.419376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:02.419376+0000","last_clean_scrub_stamp":"2023-10-11T10:43:02.419376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T20:21:46.997470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":481,"reported_epoch":467,"state":"active+clean","last_fresh":"2023-10-11T11:19:56.769946+0000","last_change":"2023-10-11T10:43:03.434569+0000","last_active":"2023-10-11T11:19:56.769946+0000","last_peered":"2023-10-11T11:19:56.769946+0000","last_clean":"2023-10-11T11:19:56.769946+0000","last_became_active":"2023-10-11T10:43:03.434147+0000","last_became_peered":"2023-10-11T10:43:03.434147+0000","last_unstale":"2023-10-11T11:19:56.769946+0000","last_undegraded":"2023-10-11T11:19:56.769946+0000","last_fullsized":"2023-10-11T11:19:56.769946+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:02.419376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:02.419376+0000","last_clean_scrub_stamp":"2023-10-11T10:43:02.419376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:12:12.693291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738850,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":151472,"kb_used_data":3748,"kb_used_omap":1,"kb_used_meta":147710,"kb_avail":94220368,"statfs":{"total":96636764160,"available":96481656832,"internally_reserved":0,"allocated":3837952,"data_stored":2177799,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":151255494},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 11:22:08 2023","interfaces":[{"interface":"back","average":{"1min":0.36099999999999999,"5min":0.34499999999999997,"15min":0.32900000000000001},"min":{"1min":0.218,"5min":0.21199999999999999,"15min":0.17499999999999999},"max":{"1min":0.53000000000000003,"5min":0.61599999999999999,"15min":1.143},"last":0.28599999999999998},{"interface":"front","average":{"1min":0.41299999999999998,"5min":0.40699999999999997,"15min":0.39000000000000001},"min":{"1min":0.32100000000000001,"5min":0.25900000000000001,"15min":0.255},"max":{"1min":0.54700000000000004,"5min":0.70799999999999996,"15min":0.78900000000000003},"last":0.38700000000000001}]},{"osd":1,"last update":"Wed Oct 11 11:22:08 2023","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.36799999999999999,"15min":0.35199999999999998},"min":{"1min":0.28299999999999997,"5min":0.25800000000000001,"15min":0.222},"max":{"1min":0.53600000000000003,"5min":0.57599999999999996,"15min":0.68700000000000006},"last":0.31},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.36599999999999999,"15min":0.34799999999999998},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.20999999999999999},"max":{"1min":0.51000000000000001,"5min":0.54100000000000004,"15min":1.2090000000000001},"last":0.34699999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738848,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":157620,"kb_used_data":4328,"kb_used_omap":1,"kb_used_meta":153278,"kb_avail":94214220,"statfs":{"total":96636764160,"available":96475361280,"internally_reserved":0,"allocated":4431872,"data_stored":2768167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":156957126},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Oct 11 11:22:14 2023","interfaces":[{"interface":"back","average":{"1min":0.32500000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"min":{"1min":0.23599999999999999,"5min":0.23300000000000001,"15min":0.183},"max":{"1min":0.45500000000000002,"5min":0.5,"15min":3.3340000000000001},"last":0.32600000000000001},{"interface":"front","average":{"1min":0.40200000000000002,"5min":0.40400000000000003,"15min":0.40500000000000003},"min":{"1min":0.311,"5min":0.28499999999999998,"15min":0.23400000000000001},"max":{"1min":0.53300000000000003,"5min":0.54200000000000004,"15min":3.3839999999999999},"last":0.41699999999999998}]},{"osd":2,"last update":"Wed Oct 11 11:22:14 2023","interfaces":[{"interface":"back","average":{"1min":0.35299999999999998,"5min":0.35199999999999998,"15min":0.34599999999999997},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.20200000000000001},"max":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":1.895},"last":0.33900000000000002},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.36799999999999999,"15min":0.36699999999999999},"min":{"1min":0.27000000000000002,"5min":0.26500000000000001,"15min":0.20699999999999999},"max":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":2.3580000000000001},"last":0.39700000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738847,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":147896,"kb_used_data":4332,"kb_used_omap":1,"kb_used_meta":143550,"kb_avail":94223944,"statfs":{"total":96636764160,"available":96485318656,"internally_reserved":0,"allocated":4435968,"data_stored":2768936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":146995654},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 11:22:14 2023","interfaces":[{"interface":"back","average":{"1min":0.376,"5min":0.35199999999999998,"15min":0.32900000000000001},"min":{"1min":0.23899999999999999,"5min":0.20399999999999999,"15min":0.19800000000000001},"max":{"1min":0.5,"5min":0.55600000000000005,"15min":0.55600000000000005},"last":0.29899999999999999},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.35899999999999999,"15min":0.33900000000000002},"min":{"1min":0.27200000000000002,"5min":0.24399999999999999,"15min":0.193},"max":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"last":0.32100000000000001}]},{"osd":2,"last update":"Wed Oct 11 11:22:14 2023","interfaces":[{"interface":"back","average":{"1min":0.39700000000000002,"5min":0.378,"15min":0.35499999999999998},"min":{"1min":0.27900000000000003,"5min":0.22,"15min":0.21199999999999999},"max":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.55300000000000005},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.39900000000000002,"5min":0.377,"15min":0.35799999999999998},"min":{"1min":0.28599999999999998,"5min":0.251,"15min":0.22500000000000001},"max":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.69899999999999995},"last":0.38500000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-11T11:22:42.686 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-10-11T10:43:02.419376+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=11, tm_hour=10, tm_min=43, tm_sec=2, tm_wday=2, tm_yday=284, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=11, tm_hour=11, tm_min=22, tm_sec=39, tm_wday=2, tm_yday=284, tm_isdst=0) 2023-10-11T11:22:42.687 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2023-10-11T10:43:02.419376+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=11, tm_hour=10, tm_min=43, tm_sec=2, tm_wday=2, tm_yday=284, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=11, tm_hour=11, tm_min=22, tm_sec=39, tm_wday=2, tm_yday=284, tm_isdst=0) 2023-10-11T11:22:42.687 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2023-10-11T10:43:02.419376+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=11, tm_hour=10, tm_min=43, tm_sec=2, tm_wday=2, tm_yday=284, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=11, tm_hour=11, tm_min=22, tm_sec=39, tm_wday=2, tm_yday=284, tm_isdst=0) 2023-10-11T11:22:42.688 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-11T10:43:02.419376+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=11, tm_hour=10, tm_min=43, tm_sec=2, tm_wday=2, tm_yday=284, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=11, tm_hour=11, tm_min=22, tm_sec=39, tm_wday=2, tm_yday=284, tm_isdst=0) 2023-10-11T11:22:42.688 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2023-10-11T10:42:56.657464+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=11, tm_hour=10, tm_min=42, tm_sec=56, tm_wday=2, tm_yday=284, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=11, tm_hour=11, tm_min=22, tm_sec=39, tm_wday=2, tm_yday=284, tm_isdst=0) 2023-10-11T11:22:42.689 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-10-11T10:43:02.419376+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=11, tm_hour=10, tm_min=43, tm_sec=2, tm_wday=2, tm_yday=284, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=11, tm_hour=11, tm_min=22, tm_sec=39, tm_wday=2, tm_yday=284, tm_isdst=0) 2023-10-11T11:22:42.689 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-11T10:43:02.419376+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=11, tm_hour=10, tm_min=43, tm_sec=2, tm_wday=2, tm_yday=284, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=11, tm_hour=11, tm_min=22, tm_sec=39, tm_wday=2, tm_yday=284, tm_isdst=0) 2023-10-11T11:22:42.689 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-11T10:43:02.419376+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=11, tm_hour=10, tm_min=43, tm_sec=2, tm_wday=2, tm_yday=284, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=11, tm_hour=11, tm_min=22, tm_sec=39, tm_wday=2, tm_yday=284, tm_isdst=0) 2023-10-11T11:22:42.690 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-10-11T10:43:02.419376+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=11, tm_hour=10, tm_min=43, tm_sec=2, tm_wday=2, tm_yday=284, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=11, tm_hour=11, tm_min=22, tm_sec=39, tm_wday=2, tm_yday=284, tm_isdst=0) 2023-10-11T11:22:42.690 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-11T11:23:02.692 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2023-10-11T11:23:03.041 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-11T11:23:03.043 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2023-10-11T11:23:03.055 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":1666,"stamp":"2023-10-11T11:23:01.108735+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":456976,"kb_used_data":12408,"kb_used_omap":4,"kb_used_meta":444539,"kb_avail":282658544,"statfs":{"total":289910292480,"available":289442349056,"internally_reserved":0,"allocated":12705792,"data_stored":7714902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4782,"internal_metadata":455208274},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002512"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":491,"reported_epoch":467,"state":"active+clean","last_fresh":"2023-10-11T11:22:53.629753+0000","last_change":"2023-10-11T11:22:53.629812+0000","last_active":"2023-10-11T11:22:53.629753+0000","last_peered":"2023-10-11T11:22:53.629753+0000","last_clean":"2023-10-11T11:22:53.629753+0000","last_became_active":"2023-10-11T10:43:03.432855+0000","last_became_peered":"2023-10-11T10:43:03.432855+0000","last_unstale":"2023-10-11T11:22:53.629753+0000","last_undegraded":"2023-10-11T11:22:53.629753+0000","last_fullsized":"2023-10-11T11:22:53.629753+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T11:22:53.629613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T11:22:53.629613+0000","last_clean_scrub_stamp":"2023-10-11T11:22:53.629613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T17:52:30.569935+0000","scrub_duration":0.0015574650000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":490,"reported_epoch":467,"state":"active+clean","last_fresh":"2023-10-11T11:22:42.372810+0000","last_change":"2023-10-11T11:22:42.372892+0000","last_active":"2023-10-11T11:22:42.372810+0000","last_peered":"2023-10-11T11:22:42.372810+0000","last_clean":"2023-10-11T11:22:42.372810+0000","last_became_active":"2023-10-11T10:43:03.434392+0000","last_became_peered":"2023-10-11T10:43:03.434392+0000","last_unstale":"2023-10-11T11:22:42.372810+0000","last_undegraded":"2023-10-11T11:22:42.372810+0000","last_fullsized":"2023-10-11T11:22:42.372810+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T11:22:42.372700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T11:22:42.372700+0000","last_clean_scrub_stamp":"2023-10-11T11:22:42.372700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T15:42:03.509506+0000","scrub_duration":0.006041416,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":491,"reported_epoch":467,"state":"active+clean","last_fresh":"2023-10-11T11:22:47.196853+0000","last_change":"2023-10-11T11:22:47.196905+0000","last_active":"2023-10-11T11:22:47.196853+0000","last_peered":"2023-10-11T11:22:47.196853+0000","last_clean":"2023-10-11T11:22:47.196853+0000","last_became_active":"2023-10-11T10:43:03.434104+0000","last_became_peered":"2023-10-11T10:43:03.434104+0000","last_unstale":"2023-10-11T11:22:47.196853+0000","last_undegraded":"2023-10-11T11:22:47.196853+0000","last_fullsized":"2023-10-11T11:22:47.196853+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T11:22:47.196732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T11:22:47.196732+0000","last_clean_scrub_stamp":"2023-10-11T11:22:47.196732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:39:21.407739+0000","scrub_duration":0.019938897000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":483,"reported_epoch":467,"state":"active+clean","last_fresh":"2023-10-11T11:22:41.787105+0000","last_change":"2023-10-11T10:43:03.433261+0000","last_active":"2023-10-11T11:22:41.787105+0000","last_peered":"2023-10-11T11:22:41.787105+0000","last_clean":"2023-10-11T11:22:41.787105+0000","last_became_active":"2023-10-11T10:43:03.432640+0000","last_became_peered":"2023-10-11T10:43:03.432640+0000","last_unstale":"2023-10-11T11:22:41.787105+0000","last_undegraded":"2023-10-11T11:22:41.787105+0000","last_fullsized":"2023-10-11T11:22:41.787105+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:02.419376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:02.419376+0000","last_clean_scrub_stamp":"2023-10-11T10:43:02.419376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"467'99","reported_seq":623,"reported_epoch":467,"state":"active+clean","last_fresh":"2023-10-11T11:22:45.735988+0000","last_change":"2023-10-11T11:22:45.736078+0000","last_active":"2023-10-11T11:22:45.735988+0000","last_peered":"2023-10-11T11:22:45.735988+0000","last_clean":"2023-10-11T11:22:45.735988+0000","last_became_active":"2023-10-11T10:42:57.664210+0000","last_became_peered":"2023-10-11T10:42:57.664210+0000","last_unstale":"2023-10-11T11:22:45.735988+0000","last_undegraded":"2023-10-11T11:22:45.735988+0000","last_fullsized":"2023-10-11T11:22:45.735988+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'99","last_scrub_stamp":"2023-10-11T11:22:45.735839+0000","last_deep_scrub":"467'99","last_deep_scrub_stamp":"2023-10-11T11:22:45.735839+0000","last_clean_scrub_stamp":"2023-10-11T11:22:45.735839+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T13:27:45.338797+0000","scrub_duration":0.0055586020000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":491,"reported_epoch":467,"state":"active+clean","last_fresh":"2023-10-11T11:22:43.244004+0000","last_change":"2023-10-11T11:22:43.244059+0000","last_active":"2023-10-11T11:22:43.244004+0000","last_peered":"2023-10-11T11:22:43.244004+0000","last_clean":"2023-10-11T11:22:43.244004+0000","last_became_active":"2023-10-11T10:43:03.434471+0000","last_became_peered":"2023-10-11T10:43:03.434471+0000","last_unstale":"2023-10-11T11:22:43.244004+0000","last_undegraded":"2023-10-11T11:22:43.244004+0000","last_fullsized":"2023-10-11T11:22:43.244004+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T11:22:43.243877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T11:22:43.243877+0000","last_clean_scrub_stamp":"2023-10-11T11:22:43.243877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T15:48:28.555244+0000","scrub_duration":0.013523881999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":483,"reported_epoch":467,"state":"active+clean","last_fresh":"2023-10-11T11:22:41.807046+0000","last_change":"2023-10-11T10:43:03.435783+0000","last_active":"2023-10-11T11:22:41.807046+0000","last_peered":"2023-10-11T11:22:41.807046+0000","last_clean":"2023-10-11T11:22:41.807046+0000","last_became_active":"2023-10-11T10:43:03.435469+0000","last_became_peered":"2023-10-11T10:43:03.435469+0000","last_unstale":"2023-10-11T11:22:41.807046+0000","last_undegraded":"2023-10-11T11:22:41.807046+0000","last_fullsized":"2023-10-11T11:22:41.807046+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T10:43:02.419376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T10:43:02.419376+0000","last_clean_scrub_stamp":"2023-10-11T10:43:02.419376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":490,"reported_epoch":467,"state":"active+clean","last_fresh":"2023-10-11T11:22:46.749447+0000","last_change":"2023-10-11T11:22:46.749528+0000","last_active":"2023-10-11T11:22:46.749447+0000","last_peered":"2023-10-11T11:22:46.749447+0000","last_clean":"2023-10-11T11:22:46.749447+0000","last_became_active":"2023-10-11T10:43:03.432238+0000","last_became_peered":"2023-10-11T10:43:03.432238+0000","last_unstale":"2023-10-11T11:22:46.749447+0000","last_undegraded":"2023-10-11T11:22:46.749447+0000","last_fullsized":"2023-10-11T11:22:46.749447+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T11:22:46.749318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T11:22:46.749318+0000","last_clean_scrub_stamp":"2023-10-11T11:22:46.749318+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T16:12:09.044068+0000","scrub_duration":0.0084423190000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":490,"reported_epoch":467,"state":"active+clean","last_fresh":"2023-10-11T11:22:50.722872+0000","last_change":"2023-10-11T11:22:50.722941+0000","last_active":"2023-10-11T11:22:50.722872+0000","last_peered":"2023-10-11T11:22:50.722872+0000","last_clean":"2023-10-11T11:22:50.722872+0000","last_became_active":"2023-10-11T10:43:03.434147+0000","last_became_peered":"2023-10-11T10:43:03.434147+0000","last_unstale":"2023-10-11T11:22:50.722872+0000","last_undegraded":"2023-10-11T11:22:50.722872+0000","last_fullsized":"2023-10-11T11:22:50.722872+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T11:22:50.722747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T11:22:50.722747+0000","last_clean_scrub_stamp":"2023-10-11T11:22:50.722747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T15:23:57.219785+0000","scrub_duration":0.026749129999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738854,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":151468,"kb_used_data":3748,"kb_used_omap":1,"kb_used_meta":147710,"kb_avail":94220372,"statfs":{"total":96636764160,"available":96481660928,"internally_reserved":0,"allocated":3837952,"data_stored":2177799,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":151255494},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 11:22:08 2023","interfaces":[{"interface":"back","average":{"1min":0.36099999999999999,"5min":0.34499999999999997,"15min":0.32900000000000001},"min":{"1min":0.218,"5min":0.21199999999999999,"15min":0.17499999999999999},"max":{"1min":0.53000000000000003,"5min":0.61599999999999999,"15min":1.143},"last":0.27300000000000002},{"interface":"front","average":{"1min":0.41299999999999998,"5min":0.40699999999999997,"15min":0.39000000000000001},"min":{"1min":0.32100000000000001,"5min":0.25900000000000001,"15min":0.255},"max":{"1min":0.54700000000000004,"5min":0.70799999999999996,"15min":0.78900000000000003},"last":0.34999999999999998}]},{"osd":1,"last update":"Wed Oct 11 11:22:08 2023","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.36799999999999999,"15min":0.35199999999999998},"min":{"1min":0.28299999999999997,"5min":0.25800000000000001,"15min":0.222},"max":{"1min":0.53600000000000003,"5min":0.57599999999999996,"15min":0.68700000000000006},"last":0.29499999999999998},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.36599999999999999,"15min":0.34799999999999998},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.20999999999999999},"max":{"1min":0.51000000000000001,"5min":0.54100000000000004,"15min":1.2090000000000001},"last":0.32400000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738852,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":157616,"kb_used_data":4328,"kb_used_omap":1,"kb_used_meta":153278,"kb_avail":94214224,"statfs":{"total":96636764160,"available":96475365376,"internally_reserved":0,"allocated":4431872,"data_stored":2768167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":156957126},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Oct 11 11:22:14 2023","interfaces":[{"interface":"back","average":{"1min":0.32500000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"min":{"1min":0.23599999999999999,"5min":0.23300000000000001,"15min":0.183},"max":{"1min":0.45500000000000002,"5min":0.5,"15min":3.3340000000000001},"last":0.27500000000000002},{"interface":"front","average":{"1min":0.40200000000000002,"5min":0.40400000000000003,"15min":0.40500000000000003},"min":{"1min":0.311,"5min":0.28499999999999998,"15min":0.23400000000000001},"max":{"1min":0.53300000000000003,"5min":0.54200000000000004,"15min":3.3839999999999999},"last":0.35999999999999999}]},{"osd":2,"last update":"Wed Oct 11 11:22:14 2023","interfaces":[{"interface":"back","average":{"1min":0.35299999999999998,"5min":0.35199999999999998,"15min":0.34599999999999997},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.20200000000000001},"max":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":1.895},"last":0.29699999999999999},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.36799999999999999,"15min":0.36699999999999999},"min":{"1min":0.27000000000000002,"5min":0.26500000000000001,"15min":0.20699999999999999},"max":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":2.3580000000000001},"last":0.34000000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738851,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":147892,"kb_used_data":4332,"kb_used_omap":1,"kb_used_meta":143550,"kb_avail":94223948,"statfs":{"total":96636764160,"available":96485322752,"internally_reserved":0,"allocated":4435968,"data_stored":2768936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":146995654},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 11:22:14 2023","interfaces":[{"interface":"back","average":{"1min":0.376,"5min":0.35199999999999998,"15min":0.32900000000000001},"min":{"1min":0.23899999999999999,"5min":0.20399999999999999,"15min":0.19800000000000001},"max":{"1min":0.5,"5min":0.55600000000000005,"15min":0.55600000000000005},"last":0.32100000000000001},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.35899999999999999,"15min":0.33900000000000002},"min":{"1min":0.27200000000000002,"5min":0.24399999999999999,"15min":0.193},"max":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"last":0.375}]},{"osd":2,"last update":"Wed Oct 11 11:22:14 2023","interfaces":[{"interface":"back","average":{"1min":0.39700000000000002,"5min":0.378,"15min":0.35499999999999998},"min":{"1min":0.27900000000000003,"5min":0.22,"15min":0.21199999999999999},"max":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.55300000000000005},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.39900000000000002,"5min":0.377,"15min":0.35799999999999998},"min":{"1min":0.28599999999999998,"5min":0.251,"15min":0.22500000000000001},"max":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.69899999999999995},"last":0.35199999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-11T11:23:03.056 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-11T10:43:02.419376+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=11, tm_hour=10, tm_min=43, tm_sec=2, tm_wday=2, tm_yday=284, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=11, tm_hour=11, tm_min=22, tm_sec=39, tm_wday=2, tm_yday=284, tm_isdst=0) 2023-10-11T11:23:03.057 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-11T10:43:02.419376+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=11, tm_hour=10, tm_min=43, tm_sec=2, tm_wday=2, tm_yday=284, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=11, tm_hour=11, tm_min=22, tm_sec=39, tm_wday=2, tm_yday=284, tm_isdst=0) 2023-10-11T11:23:03.057 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-11T11:23:23.058 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2023-10-11T11:23:23.407 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-11T11:23:23.410 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2023-10-11T11:23:23.421 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":1677,"stamp":"2023-10-11T11:23:23.112883+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":456976,"kb_used_data":12408,"kb_used_omap":4,"kb_used_meta":444539,"kb_avail":282658544,"statfs":{"total":289910292480,"available":289442349056,"internally_reserved":0,"allocated":12705792,"data_stored":7714902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4782,"internal_metadata":455208274},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002209"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":491,"reported_epoch":467,"state":"active+clean","last_fresh":"2023-10-11T11:22:53.629753+0000","last_change":"2023-10-11T11:22:53.629812+0000","last_active":"2023-10-11T11:22:53.629753+0000","last_peered":"2023-10-11T11:22:53.629753+0000","last_clean":"2023-10-11T11:22:53.629753+0000","last_became_active":"2023-10-11T10:43:03.432855+0000","last_became_peered":"2023-10-11T10:43:03.432855+0000","last_unstale":"2023-10-11T11:22:53.629753+0000","last_undegraded":"2023-10-11T11:22:53.629753+0000","last_fullsized":"2023-10-11T11:22:53.629753+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T11:22:53.629613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T11:22:53.629613+0000","last_clean_scrub_stamp":"2023-10-11T11:22:53.629613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T17:52:30.569935+0000","scrub_duration":0.0015574650000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":490,"reported_epoch":467,"state":"active+clean","last_fresh":"2023-10-11T11:22:42.372810+0000","last_change":"2023-10-11T11:22:42.372892+0000","last_active":"2023-10-11T11:22:42.372810+0000","last_peered":"2023-10-11T11:22:42.372810+0000","last_clean":"2023-10-11T11:22:42.372810+0000","last_became_active":"2023-10-11T10:43:03.434392+0000","last_became_peered":"2023-10-11T10:43:03.434392+0000","last_unstale":"2023-10-11T11:22:42.372810+0000","last_undegraded":"2023-10-11T11:22:42.372810+0000","last_fullsized":"2023-10-11T11:22:42.372810+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T11:22:42.372700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T11:22:42.372700+0000","last_clean_scrub_stamp":"2023-10-11T11:22:42.372700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T15:42:03.509506+0000","scrub_duration":0.006041416,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":491,"reported_epoch":467,"state":"active+clean","last_fresh":"2023-10-11T11:22:47.196853+0000","last_change":"2023-10-11T11:22:47.196905+0000","last_active":"2023-10-11T11:22:47.196853+0000","last_peered":"2023-10-11T11:22:47.196853+0000","last_clean":"2023-10-11T11:22:47.196853+0000","last_became_active":"2023-10-11T10:43:03.434104+0000","last_became_peered":"2023-10-11T10:43:03.434104+0000","last_unstale":"2023-10-11T11:22:47.196853+0000","last_undegraded":"2023-10-11T11:22:47.196853+0000","last_fullsized":"2023-10-11T11:22:47.196853+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T11:22:47.196732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T11:22:47.196732+0000","last_clean_scrub_stamp":"2023-10-11T11:22:47.196732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:39:21.407739+0000","scrub_duration":0.019938897000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":491,"reported_epoch":467,"state":"active+clean","last_fresh":"2023-10-11T11:23:00.646183+0000","last_change":"2023-10-11T11:23:00.646238+0000","last_active":"2023-10-11T11:23:00.646183+0000","last_peered":"2023-10-11T11:23:00.646183+0000","last_clean":"2023-10-11T11:23:00.646183+0000","last_became_active":"2023-10-11T10:43:03.432640+0000","last_became_peered":"2023-10-11T10:43:03.432640+0000","last_unstale":"2023-10-11T11:23:00.646183+0000","last_undegraded":"2023-10-11T11:23:00.646183+0000","last_fullsized":"2023-10-11T11:23:00.646183+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T11:23:00.646052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T11:23:00.646052+0000","last_clean_scrub_stamp":"2023-10-11T11:23:00.646052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T12:20:36.394881+0000","scrub_duration":0.021388595,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"467'99","reported_seq":624,"reported_epoch":467,"state":"active+clean","last_fresh":"2023-10-11T11:22:45.736078+0000","last_change":"2023-10-11T11:22:45.736078+0000","last_active":"2023-10-11T11:22:45.736078+0000","last_peered":"2023-10-11T11:22:45.736078+0000","last_clean":"2023-10-11T11:22:45.736078+0000","last_became_active":"2023-10-11T10:42:57.664210+0000","last_became_peered":"2023-10-11T10:42:57.664210+0000","last_unstale":"2023-10-11T11:22:45.736078+0000","last_undegraded":"2023-10-11T11:22:45.736078+0000","last_fullsized":"2023-10-11T11:22:45.736078+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'99","last_scrub_stamp":"2023-10-11T11:22:45.735839+0000","last_deep_scrub":"467'99","last_deep_scrub_stamp":"2023-10-11T11:22:45.735839+0000","last_clean_scrub_stamp":"2023-10-11T11:22:45.735839+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T13:27:45.338797+0000","scrub_duration":0.0055586020000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":492,"reported_epoch":467,"state":"active+clean","last_fresh":"2023-10-11T11:22:43.244059+0000","last_change":"2023-10-11T11:22:43.244059+0000","last_active":"2023-10-11T11:22:43.244059+0000","last_peered":"2023-10-11T11:22:43.244059+0000","last_clean":"2023-10-11T11:22:43.244059+0000","last_became_active":"2023-10-11T10:43:03.434471+0000","last_became_peered":"2023-10-11T10:43:03.434471+0000","last_unstale":"2023-10-11T11:22:43.244059+0000","last_undegraded":"2023-10-11T11:22:43.244059+0000","last_fullsized":"2023-10-11T11:22:43.244059+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T11:22:43.243877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T11:22:43.243877+0000","last_clean_scrub_stamp":"2023-10-11T11:22:43.243877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T15:48:28.555244+0000","scrub_duration":0.013523881999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":491,"reported_epoch":467,"state":"active+clean","last_fresh":"2023-10-11T11:22:59.599908+0000","last_change":"2023-10-11T11:22:59.599998+0000","last_active":"2023-10-11T11:22:59.599908+0000","last_peered":"2023-10-11T11:22:59.599908+0000","last_clean":"2023-10-11T11:22:59.599908+0000","last_became_active":"2023-10-11T10:43:03.435469+0000","last_became_peered":"2023-10-11T10:43:03.435469+0000","last_unstale":"2023-10-11T11:22:59.599908+0000","last_undegraded":"2023-10-11T11:22:59.599908+0000","last_fullsized":"2023-10-11T11:22:59.599908+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T11:22:59.599757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T11:22:59.599757+0000","last_clean_scrub_stamp":"2023-10-11T11:22:59.599757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T20:40:12.985557+0000","scrub_duration":0.019546826999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":490,"reported_epoch":467,"state":"active+clean","last_fresh":"2023-10-11T11:22:46.749447+0000","last_change":"2023-10-11T11:22:46.749528+0000","last_active":"2023-10-11T11:22:46.749447+0000","last_peered":"2023-10-11T11:22:46.749447+0000","last_clean":"2023-10-11T11:22:46.749447+0000","last_became_active":"2023-10-11T10:43:03.432238+0000","last_became_peered":"2023-10-11T10:43:03.432238+0000","last_unstale":"2023-10-11T11:22:46.749447+0000","last_undegraded":"2023-10-11T11:22:46.749447+0000","last_fullsized":"2023-10-11T11:22:46.749447+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T11:22:46.749318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T11:22:46.749318+0000","last_clean_scrub_stamp":"2023-10-11T11:22:46.749318+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T16:12:09.044068+0000","scrub_duration":0.0084423190000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":490,"reported_epoch":467,"state":"active+clean","last_fresh":"2023-10-11T11:22:50.722872+0000","last_change":"2023-10-11T11:22:50.722941+0000","last_active":"2023-10-11T11:22:50.722872+0000","last_peered":"2023-10-11T11:22:50.722872+0000","last_clean":"2023-10-11T11:22:50.722872+0000","last_became_active":"2023-10-11T10:43:03.434147+0000","last_became_peered":"2023-10-11T10:43:03.434147+0000","last_unstale":"2023-10-11T11:22:50.722872+0000","last_undegraded":"2023-10-11T11:22:50.722872+0000","last_fullsized":"2023-10-11T11:22:50.722872+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T11:22:50.722747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T11:22:50.722747+0000","last_clean_scrub_stamp":"2023-10-11T11:22:50.722747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T15:23:57.219785+0000","scrub_duration":0.026749129999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738859,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":151468,"kb_used_data":3748,"kb_used_omap":1,"kb_used_meta":147710,"kb_avail":94220372,"statfs":{"total":96636764160,"available":96481660928,"internally_reserved":0,"allocated":3837952,"data_stored":2177799,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":151255494},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 11:23:09 2023","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.35299999999999998,"15min":0.33200000000000002},"min":{"1min":0.183,"5min":0.183,"15min":0.17499999999999999},"max":{"1min":0.52000000000000002,"5min":0.61599999999999999,"15min":1.143},"last":0.443},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.41399999999999998,"15min":0.39300000000000002},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.255},"max":{"1min":0.56100000000000005,"5min":0.70799999999999996,"15min":0.78900000000000003},"last":0.499}]},{"osd":1,"last update":"Wed Oct 11 11:23:09 2023","interfaces":[{"interface":"back","average":{"1min":0.38800000000000001,"5min":0.373,"15min":0.35499999999999998},"min":{"1min":0.253,"5min":0.253,"15min":0.224},"max":{"1min":0.54600000000000004,"5min":0.57599999999999996,"15min":0.68700000000000006},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.376,"5min":0.37,"15min":0.34899999999999998},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.20999999999999999},"max":{"1min":0.53200000000000003,"5min":0.54100000000000004,"15min":1.2090000000000001},"last":0.48299999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738857,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":157616,"kb_used_data":4328,"kb_used_omap":1,"kb_used_meta":153278,"kb_avail":94214224,"statfs":{"total":96636764160,"available":96475365376,"internally_reserved":0,"allocated":4431872,"data_stored":2768167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":156957126},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Oct 11 11:23:16 2023","interfaces":[{"interface":"back","average":{"1min":0.32500000000000001,"5min":0.32900000000000001,"15min":0.318},"min":{"1min":0.23999999999999999,"5min":0.23300000000000001,"15min":0.183},"max":{"1min":0.432,"5min":0.5,"15min":1.962},"last":0.315},{"interface":"front","average":{"1min":0.39400000000000002,"5min":0.40200000000000002,"15min":0.39500000000000002},"min":{"1min":0.29499999999999998,"5min":0.28499999999999998,"15min":0.23400000000000001},"max":{"1min":0.504,"5min":0.54200000000000004,"15min":2.0009999999999999},"last":0.29499999999999998}]},{"osd":2,"last update":"Wed Oct 11 11:23:15 2023","interfaces":[{"interface":"back","average":{"1min":0.34100000000000003,"5min":0.35099999999999998,"15min":0.34200000000000003},"min":{"1min":0.25,"5min":0.22500000000000001,"15min":0.20699999999999999},"max":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":1.022},"last":0.33200000000000002},{"interface":"front","average":{"1min":0.39800000000000002,"5min":0.374,"15min":0.36399999999999999},"min":{"1min":0.32900000000000001,"5min":0.26500000000000001,"15min":0.20699999999999999},"max":{"1min":0.55700000000000005,"5min":0.55800000000000005,"15min":1.4399999999999999},"last":0.35699999999999998}]}]},{"osd":1,"up_from":8,"seq":34359738856,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":147892,"kb_used_data":4332,"kb_used_omap":1,"kb_used_meta":143550,"kb_avail":94223948,"statfs":{"total":96636764160,"available":96485322752,"internally_reserved":0,"allocated":4435968,"data_stored":2768936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":146995654},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 11:23:15 2023","interfaces":[{"interface":"back","average":{"1min":0.33000000000000002,"5min":0.34699999999999998,"15min":0.33000000000000002},"min":{"1min":0.20899999999999999,"5min":0.20399999999999999,"15min":0.19800000000000001},"max":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.34899999999999998,"5min":0.35799999999999998,"15min":0.34100000000000003},"min":{"1min":0.27900000000000003,"5min":0.25800000000000001,"15min":0.193},"max":{"1min":0.51300000000000001,"5min":0.54400000000000004,"15min":0.54400000000000004},"last":0.317}]},{"osd":2,"last update":"Wed Oct 11 11:23:15 2023","interfaces":[{"interface":"back","average":{"1min":0.36699999999999999,"5min":0.379,"15min":0.35699999999999998},"min":{"1min":0.253,"5min":0.253,"15min":0.21199999999999999},"max":{"1min":0.51200000000000001,"5min":0.54800000000000004,"15min":0.55300000000000005},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.36199999999999999,"5min":0.375,"15min":0.35899999999999999},"min":{"1min":0.254,"5min":0.251,"15min":0.22500000000000001},"max":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.69899999999999995},"last":0.30399999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-11T11:23:23.423 DEBUG:teuthology.orchestra.run.smithi050:> sudo ceph --cluster cluster1 config set global mon_health_to_clog false 2023-10-11T11:23:23.777 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_idryomov_ceph_3d099a3859cc211cf8dd2acdaf3194ba0e96ea9c/qa/tasks/ceph.py", line 1940, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_idryomov_ceph_3d099a3859cc211cf8dd2acdaf3194ba0e96ea9c/qa/tasks/ceph.py", line 1950, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_idryomov_ceph_3d099a3859cc211cf8dd2acdaf3194ba0e96ea9c/qa/tasks/ceph_manager.py", line 2743, in wait_for_clean while not self.is_clean(): File "/home/teuthworker/src/github.com_idryomov_ceph_3d099a3859cc211cf8dd2acdaf3194ba0e96ea9c/qa/tasks/ceph_manager.py", line 2679, in is_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_idryomov_ceph_3d099a3859cc211cf8dd2acdaf3194ba0e96ea9c/qa/tasks/ceph_manager.py", line 2340, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_idryomov_ceph_3d099a3859cc211cf8dd2acdaf3194ba0e96ea9c/qa/tasks/ceph_manager.py", line 1595, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_idryomov_ceph_3d099a3859cc211cf8dd2acdaf3194ba0e96ea9c/qa/tasks/ceph_manager.py", line 1586, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi111 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json' 2023-10-11T11:23:23.778 INFO:teuthology.misc:Shutting down mds daemons... 2023-10-11T11:23:23.778 INFO:teuthology.misc:Shutting down osd daemons... 2023-10-11T11:23:23.779 DEBUG:tasks.ceph.cluster1.osd.0:waiting for process to exit 2023-10-11T11:23:23.779 INFO:teuthology.orchestra.run:waiting for 300 2023-10-11T11:23:23.830 INFO:tasks.ceph.cluster1.osd.0:Stopped 2023-10-11T11:23:23.831 DEBUG:tasks.ceph.cluster1.osd.1:waiting for process to exit 2023-10-11T11:23:23.832 INFO:teuthology.orchestra.run:waiting for 300 2023-10-11T11:23:23.892 INFO:tasks.ceph.cluster1.osd.1:Stopped 2023-10-11T11:23:23.893 DEBUG:tasks.ceph.cluster1.osd.2:waiting for process to exit 2023-10-11T11:23:23.894 INFO:teuthology.orchestra.run:waiting for 300 2023-10-11T11:23:23.938 INFO:tasks.ceph.cluster1.osd.2:Stopped 2023-10-11T11:23:23.939 INFO:teuthology.misc:Shutting down mgr daemons... 2023-10-11T11:23:23.939 DEBUG:tasks.ceph.cluster1.mgr.x:waiting for process to exit 2023-10-11T11:23:23.940 INFO:teuthology.orchestra.run:waiting for 300 2023-10-11T11:23:23.965 INFO:tasks.ceph.cluster1.mgr.x:Stopped 2023-10-11T11:23:23.966 INFO:teuthology.misc:Shutting down mon daemons... 2023-10-11T11:23:23.966 DEBUG:tasks.ceph.cluster1.mon.a:waiting for process to exit 2023-10-11T11:23:23.967 INFO:teuthology.orchestra.run:waiting for 300 2023-10-11T11:23:24.018 INFO:tasks.ceph.cluster1.mon.a:Stopped 2023-10-11T11:23:24.018 INFO:tasks.ceph:Checking cluster log for badness... 2023-10-11T11:23:24.019 DEBUG:teuthology.orchestra.run.smithi050:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/cluster1.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | head -n 1 2023-10-11T11:23:24.088 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster1-0 on ubuntu@smithi050.front.sepia.ceph.com 2023-10-11T11:23:24.088 DEBUG:teuthology.orchestra.run.smithi050:> sync && sudo umount -f /var/lib/ceph/osd/cluster1-0 2023-10-11T11:23:24.302 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster1-1 on ubuntu@smithi050.front.sepia.ceph.com 2023-10-11T11:23:24.303 DEBUG:teuthology.orchestra.run.smithi050:> sync && sudo umount -f /var/lib/ceph/osd/cluster1-1 2023-10-11T11:23:24.402 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster1-2 on ubuntu@smithi050.front.sepia.ceph.com 2023-10-11T11:23:24.402 DEBUG:teuthology.orchestra.run.smithi050:> sync && sudo umount -f /var/lib/ceph/osd/cluster1-2 2023-10-11T11:23:24.522 INFO:tasks.ceph:Archiving mon data... 2023-10-11T11:23:24.523 DEBUG:teuthology.misc:Transferring archived files from smithi050:/var/lib/ceph/mon/cluster1-a to /home/teuthworker/archive/dis-2023-10-11_10:22:06-rbd-wip-dis-testing-distro-default-smithi/7421102/data/cluster1.mon.a.tgz 2023-10-11T11:23:24.524 DEBUG:teuthology.orchestra.run.smithi050:> mktemp 2023-10-11T11:23:24.542 INFO:teuthology.orchestra.run.smithi050.stdout:/tmp/tmp.iCsYOl88Pg 2023-10-11T11:23:24.543 DEBUG:teuthology.orchestra.run.smithi050:> sudo tar cz -f - -C /var/lib/ceph/mon/cluster1-a -- . > /tmp/tmp.iCsYOl88Pg 2023-10-11T11:23:24.697 DEBUG:teuthology.orchestra.run.smithi050:> sudo chmod 0666 /tmp/tmp.iCsYOl88Pg 2023-10-11T11:23:24.785 DEBUG:teuthology.orchestra.remote:smithi050:/tmp/tmp.iCsYOl88Pg is 351KB 2023-10-11T11:23:24.880 DEBUG:teuthology.orchestra.run.smithi050:> rm -fr /tmp/tmp.iCsYOl88Pg 2023-10-11T11:23:24.898 INFO:tasks.ceph:Cleaning ceph cluster... 2023-10-11T11:23:24.898 DEBUG:teuthology.orchestra.run.smithi050:> sudo rm -rf -- /etc/ceph/cluster1.conf /etc/ceph/cluster1.keyring /home/ubuntu/cephtest/cluster1.data /home/ubuntu/cephtest/cluster1.monmap /home/ubuntu/cephtest/../*.pid 2023-10-11T11:23:24.940 DEBUG:teuthology.orchestra.run.smithi111:> sudo rm -rf -- /etc/ceph/cluster1.conf /etc/ceph/cluster1.keyring /home/ubuntu/cephtest/cluster1.data /home/ubuntu/cephtest/cluster1.monmap /home/ubuntu/cephtest/../*.pid 2023-10-11T11:23:24.974 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2023-10-11T11:23:24.975 DEBUG:teuthology.orchestra.run.smithi050:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-10-11T11:23:25.008 DEBUG:teuthology.orchestra.run.smithi111:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-10-11T11:23:25.052 INFO:teuthology.orchestra.run.smithi050.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-10-11T11:23:25.105 INFO:teuthology.orchestra.run.smithi111.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-10-11T11:23:25.112 INFO:tasks.ceph:Archiving crash dumps... 2023-10-11T11:23:25.114 DEBUG:teuthology.misc:Transferring archived files from smithi050:/var/lib/ceph/crash to /home/teuthworker/archive/dis-2023-10-11_10:22:06-rbd-wip-dis-testing-distro-default-smithi/7421102/remote/smithi050/crash 2023-10-11T11:23:25.115 DEBUG:teuthology.orchestra.run.smithi050:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-10-11T11:23:25.143 DEBUG:teuthology.misc:Transferring archived files from smithi111:/var/lib/ceph/crash to /home/teuthworker/archive/dis-2023-10-11_10:22:06-rbd-wip-dis-testing-distro-default-smithi/7421102/remote/smithi111/crash 2023-10-11T11:23:25.145 DEBUG:teuthology.orchestra.run.smithi111:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-10-11T11:23:25.222 INFO:tasks.ceph:Compressing logs... 2023-10-11T11:23:25.223 DEBUG:teuthology.orchestra.run.smithi050:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-10-11T11:23:25.225 DEBUG:teuthology.orchestra.run.smithi111:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-10-11T11:23:25.250 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.104427.log 2023-10-11T11:23:25.258 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.100176.log 2023-10-11T11:23:25.259 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/cluster1-client.admin.104427.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.104290.log 2023-10-11T11:23:25.259 INFO:teuthology.orchestra.run.smithi050.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.104427.log.gz 2023-10-11T11:23:25.259 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.100103.log 2023-10-11T11:23:25.259 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/cluster1-client.admin.100176.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.100176.log.gz 2023-10-11T11:23:25.260 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/cluster1-client.admin.104290.log: gzip -5 --verbose -- /var/log/ceph/cluster1-osd.0.log 2023-10-11T11:23:25.260 INFO:teuthology.orchestra.run.smithi050.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.104290.log.gz 2023-10-11T11:23:25.260 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/cluster1-client.admin.100103.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.104658.log 2023-10-11T11:23:25.260 INFO:teuthology.orchestra.run.smithi050.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.100103.log.gz 2023-10-11T11:23:25.260 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/cluster1-osd.0.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.99895.log 2023-10-11T11:23:25.261 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.104704.log 2023-10-11T11:23:25.261 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/cluster1-client.admin.104658.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.104658.log.gz 2023-10-11T11:23:25.261 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/cluster1-client.admin.99895.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.99942.log 2023-10-11T11:23:25.261 INFO:teuthology.orchestra.run.smithi050.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.99895.log.gz 2023-10-11T11:23:25.261 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/cluster1-client.admin.104704.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.106724.log 2023-10-11T11:23:25.262 INFO:teuthology.orchestra.run.smithi050.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.104704.log.gz 2023-10-11T11:23:25.262 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/cluster1-client.admin.99942.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.99942.log.gz 2023-10-11T11:23:25.262 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.104565.log 2023-10-11T11:23:25.262 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/cluster1-client.admin.106724.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.107184.log 2023-10-11T11:23:25.262 INFO:teuthology.orchestra.run.smithi050.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.106724.log.gz 2023-10-11T11:23:25.263 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/cluster1-client.admin.104565.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.99989.log 2023-10-11T11:23:25.263 INFO:teuthology.orchestra.run.smithi050.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.104565.log.gz 2023-10-11T11:23:25.263 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/cluster1-client.admin.107184.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.104934.log 2023-10-11T11:23:25.263 INFO:teuthology.orchestra.run.smithi050.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.107184.log.gz 2023-10-11T11:23:25.263 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/cluster1-client.admin.99989.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.106908.log 2023-10-11T11:23:25.264 INFO:teuthology.orchestra.run.smithi050.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.99989.log.gz 2023-10-11T11:23:25.264 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/cluster1-client.admin.104934.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.95062.log 2023-10-11T11:23:25.264 INFO:teuthology.orchestra.run.smithi050.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.104934.log.gz 2023-10-11T11:23:25.264 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/cluster1-client.admin.106908.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.104336.log 2023-10-11T11:23:25.264 INFO:teuthology.orchestra.run.smithi050.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.106908.log.gz 2023-10-11T11:23:25.265 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph.tmp-client.admin.95062.log: 0.0%gzip -- replaced with /var/log/ceph/ceph.tmp-client.admin.95062.log.gz-5 2023-10-11T11:23:25.265 INFO:teuthology.orchestra.run.smithi050.stderr:--verbose -- /var/log/ceph/cluster2-mgr.x.log 2023-10-11T11:23:25.265 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/cluster1-client.admin.104336.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.104796.log 2023-10-11T11:23:25.265 INFO:teuthology.orchestra.run.smithi050.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.104336.log.gz 2023-10-11T11:23:25.265 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/cluster2-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.104980.log 2023-10-11T11:23:25.265 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/cluster1-client.admin.104796.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.104796.log.gz 2023-10-11T11:23:25.266 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.104152.log 2023-10-11T11:23:25.266 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.104703.log 2023-10-11T11:23:25.267 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/cluster1-client.admin.104980.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.104980.log.gz 2023-10-11T11:23:25.267 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/cluster1-client.admin.104152.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.106862.log 2023-10-11T11:23:25.267 INFO:teuthology.orchestra.run.smithi050.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.104152.log.gz 2023-10-11T11:23:25.267 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/cluster1-client.admin.104703.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.104244.log 2023-10-11T11:23:25.267 INFO:teuthology.orchestra.run.smithi050.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.104703.log.gz 2023-10-11T11:23:25.267 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/cluster1-client.admin.106862.log: gzip -5 --verbose -- /var/log/ceph/cluster1-osd.1.log 2023-10-11T11:23:25.268 INFO:teuthology.orchestra.run.smithi050.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.106862.log.gz 2023-10-11T11:23:25.268 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/cluster1-client.admin.104244.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.104244.log.gzgzip 2023-10-11T11:23:25.268 INFO:teuthology.orchestra.run.smithi050.stderr:-5 --verbose -- /var/log/ceph/cluster1-mon.a.log 2023-10-11T11:23:25.268 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/cluster1-osd.1.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.107000.log 2023-10-11T11:23:25.272 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/cluster1-mon.a.log: /var/log/ceph/cluster1-client.admin.107000.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.107000.log.gz 2023-10-11T11:23:25.272 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.107046.log 2023-10-11T11:23:25.272 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.105026.log 2023-10-11T11:23:25.273 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/cluster1-client.admin.107046.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.107046.log.gz 2023-10-11T11:23:25.273 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/cluster1-client.admin.105026.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.105026.log.gz 2023-10-11T11:23:25.273 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.104842.log 2023-10-11T11:23:25.273 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1.log 2023-10-11T11:23:25.273 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/cluster1-client.admin.104842.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.104842.log.gz 2023-10-11T11:23:25.274 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/cluster1.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.106770.log 2023-10-11T11:23:25.274 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/cluster1-client.admin.106770.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.106770.log.gz 2023-10-11T11:23:25.274 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.107092.log 2023-10-11T11:23:25.274 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/cluster1-client.admin.107092.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.107092.log.gz 2023-10-11T11:23:25.274 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.107328.log 2023-10-11T11:23:25.275 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/cluster1-client.admin.107328.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.107328.log.gz 2023-10-11T11:23:25.275 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.104198.log 2023-10-11T11:23:25.275 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/cluster1-client.admin.104198.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.104198.log.gz 2023-10-11T11:23:25.277 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1.audit.log 2023-10-11T11:23:25.278 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/cluster1.audit.log: 88.0% -- replaced with /var/log/ceph/cluster1.log.gz 2023-10-11T11:23:25.278 INFO:teuthology.orchestra.run.smithi050.stderr: 92.9% -- replaced with /var/log/ceph/cluster1.audit.log.gz 2023-10-11T11:23:25.278 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.106816.log 2023-10-11T11:23:25.278 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-osd.2.log 2023-10-11T11:23:25.278 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/cluster1-client.admin.106816.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.106816.log.gz 2023-10-11T11:23:25.279 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/cluster1-osd.2.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.104520.log 2023-10-11T11:23:25.279 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/cluster1-client.admin.104520.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.104520.log.gz 2023-10-11T11:23:25.279 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.99723.log 2023-10-11T11:23:25.283 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/cluster1-client.admin.99723.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.99723.log.gz 2023-10-11T11:23:25.283 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.104001.log 2023-10-11T11:23:25.284 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/cluster1-client.admin.104001.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.104001.log.gz 2023-10-11T11:23:25.284 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.106954.log 2023-10-11T11:23:25.284 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/cluster1-client.admin.106954.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.104819.log 2023-10-11T11:23:25.285 INFO:teuthology.orchestra.run.smithi050.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.106954.log.gz 2023-10-11T11:23:25.285 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.104089.log 2023-10-11T11:23:25.285 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/cluster1-client.admin.104819.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.104819.log.gz 2023-10-11T11:23:25.285 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/cluster1-client.admin.104089.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.104089.log.gz 2023-10-11T11:23:25.285 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.104428.log 2023-10-11T11:23:25.286 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/cluster1-client.admin.104428.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.104566.log 2023-10-11T11:23:25.286 INFO:teuthology.orchestra.run.smithi050.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.104428.log.gz 2023-10-11T11:23:25.286 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-mgr.x.log 2023-10-11T11:23:25.286 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/cluster1-client.admin.104566.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.104566.log.gz 2023-10-11T11:23:25.286 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.104426.log 2023-10-11T11:23:25.287 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/cluster1-mgr.x.log: /var/log/ceph/cluster1-client.admin.104426.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.104426.log.gz 2023-10-11T11:23:25.287 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.107138.log 2023-10-11T11:23:25.287 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/cluster1-client.admin.107138.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.107138.log.gz 2023-10-11T11:23:25.287 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.107230.log 2023-10-11T11:23:25.288 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/cluster1-client.admin.107230.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.107230.log.gz 2023-10-11T11:23:25.288 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.107284.log 2023-10-11T11:23:25.288 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/cluster1-client.admin.107284.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.107284.log.gz 2023-10-11T11:23:25.293 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.107546.log 2023-10-11T11:23:25.293 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.120899.log 2023-10-11T11:23:25.294 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.107546.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118154.log 2023-10-11T11:23:25.294 INFO:teuthology.orchestra.run.smithi111.stderr: 54.2% -- replaced with /var/log/ceph/cluster2-client.mirror.107546.log.gz 2023-10-11T11:23:25.294 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.120899.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.114283.log 2023-10-11T11:23:25.294 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.118154.log: gzip 87.4%-5 -- replaced with /var/log/ceph/cluster2-client.mirror.120899.log.gz--verbose 2023-10-11T11:23:25.294 INFO:teuthology.orchestra.run.smithi111.stderr:-- /var/log/ceph/cluster1-client.mirror.0.131538.log 2023-10-11T11:23:25.295 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.114283.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.129058.log 2023-10-11T11:23:25.295 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.142088.log 2023-10-11T11:23:25.295 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.0.131538.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.114283.log.gz 2023-10-11T11:23:25.295 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.129058.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117830.log 2023-10-11T11:23:25.295 INFO:teuthology.orchestra.run.smithi111.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.129058.log.gz 2023-10-11T11:23:25.296 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.142088.log: gzip -5 --verbose -- /var/log/ceph/cluster2-osd.0.log 2023-10-11T11:23:25.296 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.117830.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120435.log 2023-10-11T11:23:25.296 INFO:teuthology.orchestra.run.smithi111.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.142088.log.gz 2023-10-11T11:23:25.296 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-osd.0.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.109418.log 2023-10-11T11:23:25.296 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.120435.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.146203.log 2023-10-11T11:23:25.296 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.109418.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.120435.log.gz 2023-10-11T11:23:25.297 INFO:teuthology.orchestra.run.smithi111.stderr: 87.9%/var/log/ceph/cluster2-client.admin.146203.log: -- replaced with /var/log/ceph/cluster2-client.mirror.109418.log.gz 2023-10-11T11:23:25.297 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.146203.log.gz 2023-10-11T11:23:25.297 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136365.log 2023-10-11T11:23:25.297 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120411.log 2023-10-11T11:23:25.298 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.136365.log: 85.2% -- replaced with /var/log/ceph/cluster1-client.mirror.136365.log.gz 2023-10-11T11:23:25.298 INFO:teuthology.orchestra.run.smithi111.stderr: 93.2%/var/log/ceph/cluster1-client.mirror.120411.log: -- replaced with /var/log/ceph/cluster1-client.mirror.0.131538.log.gz 2023-10-11T11:23:25.298 INFO:teuthology.orchestra.run.smithi111.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.120411.log.gz 2023-10-11T11:23:25.298 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.115413.log 2023-10-11T11:23:25.299 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.135684.log 2023-10-11T11:23:25.299 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.115413.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.108684.log 2023-10-11T11:23:25.299 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.135684.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118697.log 2023-10-11T11:23:25.299 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.108684.log: /var/log/ceph/cluster2-client.mirror.118697.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.114632.log 2023-10-11T11:23:25.299 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.114632.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.135684.log.gz 2023-10-11T11:23:25.300 INFO:teuthology.orchestra.run.smithi111.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.114632.log.gz 2023-10-11T11:23:25.300 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.128590.log 2023-10-11T11:23:25.300 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.128590.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.128590.log.gz 2023-10-11T11:23:25.300 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118655.log 2023-10-11T11:23:25.300 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.118655.log: 93.1% -- replaced with /var/log/ceph/cluster2-client.mirror.108684.log.gz 2023-10-11T11:23:25.301 INFO:teuthology.orchestra.run.smithi111.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.118655.log.gz 2023-10-11T11:23:25.301 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.139867.log 2023-10-11T11:23:25.301 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137333.log 2023-10-11T11:23:25.301 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.139867.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118754.log 2023-10-11T11:23:25.301 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.137333.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.139867.log.gz 2023-10-11T11:23:25.302 INFO:teuthology.orchestra.run.smithi111.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.137333.log.gz 2023-10-11T11:23:25.304 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126883.log 2023-10-11T11:23:25.305 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.118754.log: 87.5%/var/log/ceph/cluster1-client.mirror.126883.log: -- replaced with /var/log/ceph/cluster2-client.mirror.118754.log.gz 2023-10-11T11:23:25.305 INFO:teuthology.orchestra.run.smithi111.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.126883.log.gz 2023-10-11T11:23:25.305 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121387.log 2023-10-11T11:23:25.305 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.121387.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.121387.log.gz 2023-10-11T11:23:25.305 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123773.log 2023-10-11T11:23:25.306 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.145189.log 2023-10-11T11:23:25.306 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.123773.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.123773.log.gz 2023-10-11T11:23:25.306 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.admin.145189.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.145189.log.gz 2023-10-11T11:23:25.306 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.146111.log 2023-10-11T11:23:25.306 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122214.log 2023-10-11T11:23:25.307 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.admin.146111.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.146111.log.gz 2023-10-11T11:23:25.307 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.122214.log: 92.1% -- replaced with /var/log/ceph/cluster1-client.mirror.122214.log.gz 2023-10-11T11:23:25.307 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.133927.log 2023-10-11T11:23:25.307 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.113435.log 2023-10-11T11:23:25.307 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.133927.log: /var/log/ceph/cluster2-client.mirror.113435.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.133927.log.gz 2023-10-11T11:23:25.308 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.121363.log 2023-10-11T11:23:25.308 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.121363.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.121363.log.gz 2023-10-11T11:23:25.309 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.112844.log 2023-10-11T11:23:25.311 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.112844.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.113435.log.gz 2023-10-11T11:23:25.311 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117132.log 2023-10-11T11:23:25.311 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.117132.log: 91.6% -- replaced with /var/log/ceph/cluster2-client.mirror.112844.log.gz 2023-10-11T11:23:25.311 INFO:teuthology.orchestra.run.smithi111.stderr: 80.2% -- replaced with /var/log/ceph/cluster2-client.mirror.117132.log.gz 2023-10-11T11:23:25.311 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.135602.log 2023-10-11T11:23:25.312 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120691.log 2023-10-11T11:23:25.312 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.135602.log: /var/log/ceph/cluster1-client.mirror.120691.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.135602.log.gz 2023-10-11T11:23:25.312 INFO:teuthology.orchestra.run.smithi111.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.120691.log.gz 2023-10-11T11:23:25.312 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.130563.log 2023-10-11T11:23:25.313 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.140374.log 2023-10-11T11:23:25.314 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.130563.log: /var/log/ceph/cluster2-client.mirror.140374.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.140374.log.gz 2023-10-11T11:23:25.314 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124201.log 2023-10-11T11:23:25.314 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.124201.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.130563.log.gz 2023-10-11T11:23:25.314 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.135560.log 2023-10-11T11:23:25.315 INFO:teuthology.orchestra.run.smithi111.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.124201.log.gz 2023-10-11T11:23:25.315 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.135560.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.132560.log 2023-10-11T11:23:25.315 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.132560.log: 92.3% -- replaced with /var/log/ceph/cluster2-client.mirror.132560.log.gz 2023-10-11T11:23:25.316 INFO:teuthology.orchestra.run.smithi111.stderr: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.135560.log.gz 2023-10-11T11:23:25.316 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127609.log 2023-10-11T11:23:25.316 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.139781.log 2023-10-11T11:23:25.317 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.127609.log: /var/log/ceph/cluster2-client.mirror.139781.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.127609.log.gz 2023-10-11T11:23:25.317 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.134873.log 2023-10-11T11:23:25.318 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.134873.log: 91.3% -- replaced with /var/log/ceph/cluster2-client.mirror.139781.log.gz 2023-10-11T11:23:25.318 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.142302.log 2023-10-11T11:23:25.318 INFO:teuthology.orchestra.run.smithi111.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.134873.log.gz 2023-10-11T11:23:25.318 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.129154.log 2023-10-11T11:23:25.320 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.142302.log: 80.2% -- replaced with /var/log/ceph/cluster2-client.mirror.142302.log.gz 2023-10-11T11:23:25.320 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.129154.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.105486.log 2023-10-11T11:23:25.320 INFO:teuthology.orchestra.run.smithi111.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.129154.log.gz 2023-10-11T11:23:25.320 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.admin.105486.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.105486.log.gz 2023-10-11T11:23:25.320 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.130220.log 2023-10-11T11:23:25.321 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118033.log 2023-10-11T11:23:25.321 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.130220.log: /var/log/ceph/cluster2-client.mirror.118033.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.130220.log.gz 2023-10-11T11:23:25.322 INFO:teuthology.orchestra.run.smithi111.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.118033.log.gz 2023-10-11T11:23:25.322 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117571.log 2023-10-11T11:23:25.322 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.133484.log 2023-10-11T11:23:25.323 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.117571.log: 83.9% -- replaced with /var/log/ceph/cluster2-client.mirror.117571.log.gz 2023-10-11T11:23:25.323 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.133484.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.133484.log.gz 2023-10-11T11:23:25.324 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.132150.log 2023-10-11T11:23:25.324 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122589.log 2023-10-11T11:23:25.324 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.132150.log: /var/log/ceph/cluster1-client.mirror.122589.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.122589.log.gz 2023-10-11T11:23:25.325 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.141662.log 2023-10-11T11:23:25.325 INFO:teuthology.orchestra.run.smithi111.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.132150.log.gz 2023-10-11T11:23:25.326 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112558.log 2023-10-11T11:23:25.326 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.141662.log: /var/log/ceph/cluster1-client.mirror.112558.log: 89.1% -- replaced with /var/log/ceph/cluster1-client.mirror.112558.log.gz 2023-10-11T11:23:25.326 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.111759.log 2023-10-11T11:23:25.327 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.admin.111759.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.admin.111759.log.gz 2023-10-11T11:23:25.327 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.107461.log 2023-10-11T11:23:25.328 INFO:teuthology.orchestra.run.smithi111.stderr: 91.2% -- replaced with /var/log/ceph/cluster2-client.mirror.141662.log.gz 2023-10-11T11:23:25.328 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.107461.log: 53.5% -- replaced with /var/log/ceph/cluster2-client.mirror.107461.log.gz 2023-10-11T11:23:25.328 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122613.log 2023-10-11T11:23:25.329 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.145973.log 2023-10-11T11:23:25.330 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.122613.log: /var/log/ceph/cluster2-client.admin.145973.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.122613.log.gz 2023-10-11T11:23:25.330 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.145973.log.gz 2023-10-11T11:23:25.330 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120923.log 2023-10-11T11:23:25.330 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121728.log 2023-10-11T11:23:25.331 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.120923.log: /var/log/ceph/cluster1-client.mirror.121728.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.120923.log.gz 2023-10-11T11:23:25.331 INFO:teuthology.orchestra.run.smithi111.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.121728.log.gz 2023-10-11T11:23:25.332 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.114692.log 2023-10-11T11:23:25.332 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113908.log 2023-10-11T11:23:25.332 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.114692.log: /var/log/ceph/cluster1-client.mirror.113908.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.114692.log.gz 2023-10-11T11:23:25.334 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118334.log 2023-10-11T11:23:25.335 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.118334.log: 80.4% -- replaced with /var/log/ceph/cluster2-client.mirror.118334.log.gz 2023-10-11T11:23:25.335 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122869.log 2023-10-11T11:23:25.336 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.122869.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.122869.log.gz 2023-10-11T11:23:25.337 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137243.log 2023-10-11T11:23:25.337 INFO:teuthology.orchestra.run.smithi111.stderr: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.113908.log.gz 2023-10-11T11:23:25.338 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.128632.log 2023-10-11T11:23:25.338 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.137243.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.137243.log.gz 2023-10-11T11:23:25.339 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.128632.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.128632.log.gz 2023-10-11T11:23:25.339 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.114303.log 2023-10-11T11:23:25.339 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.134782.log 2023-10-11T11:23:25.340 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.114303.log: /var/log/ceph/cluster2-client.mirror.134782.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.114303.log.gz 2023-10-11T11:23:25.341 INFO:teuthology.orchestra.run.smithi111.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.134782.log.gz 2023-10-11T11:23:25.341 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.109690.log 2023-10-11T11:23:25.341 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.112919.log 2023-10-11T11:23:25.342 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.admin.109690.log: /var/log/ceph/cluster2-client.mirror.112919.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.109690.log.gz 2023-10-11T11:23:25.342 INFO:teuthology.orchestra.run.smithi111.stderr: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.112919.log.gz 2023-10-11T11:23:25.342 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.142808.log 2023-10-11T11:23:25.343 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137625.log 2023-10-11T11:23:25.343 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.142808.log: /var/log/ceph/cluster1-client.mirror.137625.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.137625.log.gz 2023-10-11T11:23:25.345 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110772.log 2023-10-11T11:23:25.345 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.110772.log: 91.3% -- replaced with /var/log/ceph/cluster2-client.mirror.142808.log.gz 2023-10-11T11:23:25.345 INFO:teuthology.orchestra.run.smithi111.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.110772.log.gz 2023-10-11T11:23:25.346 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.115901.log 2023-10-11T11:23:25.348 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.131268.log 2023-10-11T11:23:25.348 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.115901.log: /var/log/ceph/cluster1-client.mirror.131268.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.131268.log.gz 2023-10-11T11:23:25.348 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.112264.log 2023-10-11T11:23:25.348 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.112264.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.112264.log.gz 2023-10-11T11:23:25.349 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.144537.log 2023-10-11T11:23:25.349 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.admin.144537.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.144537.log.gz 2023-10-11T11:23:25.349 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110562.log 2023-10-11T11:23:25.349 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.110562.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.110562.log.gz 2023-10-11T11:23:25.349 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.136015.log 2023-10-11T11:23:25.350 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.136015.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.136015.log.gz 2023-10-11T11:23:25.350 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113080.log 2023-10-11T11:23:25.355 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.113080.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.113080.log.gz 2023-10-11T11:23:25.356 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123589.log 2023-10-11T11:23:25.357 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.123589.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.123589.log.gz 2023-10-11T11:23:25.357 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.140821.log 2023-10-11T11:23:25.358 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.140821.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.140821.log.gz 2023-10-11T11:23:25.358 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118833.log 2023-10-11T11:23:25.359 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.118833.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.118833.log.gz 2023-10-11T11:23:25.359 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119478.log 2023-10-11T11:23:25.370 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.119478.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.144767.log 2023-10-11T11:23:25.374 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.admin.144767.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.144767.log.gz 2023-10-11T11:23:25.376 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.114552.log 2023-10-11T11:23:25.378 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.114552.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.114552.log.gz 2023-10-11T11:23:25.378 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.107529.log 2023-10-11T11:23:25.378 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.107529.log: 57.4% -- replaced with /var/log/ceph/cluster2-client.mirror.107529.log.gz 2023-10-11T11:23:25.379 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.128069.log 2023-10-11T11:23:25.379 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.128069.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.128069.log.gz 2023-10-11T11:23:25.380 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.140309.log 2023-10-11T11:23:25.381 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.140309.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.140309.log.gz 2023-10-11T11:23:25.381 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111789.log 2023-10-11T11:23:25.382 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.111789.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.111789.log.gz 2023-10-11T11:23:25.382 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.109666.log 2023-10-11T11:23:25.394 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.109666.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.142707.log 2023-10-11T11:23:25.405 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.142707.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.142707.log.gz 2023-10-11T11:23:25.405 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.123884.log 2023-10-11T11:23:25.419 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.123884.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.122468.log 2023-10-11T11:23:25.430 INFO:teuthology.orchestra.run.smithi111.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.123884.log.gz 2023-10-11T11:23:25.432 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.122468.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.122468.log.gz 2023-10-11T11:23:25.441 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.129832.log 2023-10-11T11:23:25.453 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.129832.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.132778.log 2023-10-11T11:23:25.453 INFO:teuthology.orchestra.run.smithi111.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.129832.log.gz 2023-10-11T11:23:25.457 INFO:teuthology.orchestra.run.smithi111.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.118697.log.gz 2023-10-11T11:23:25.459 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.132778.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.132778.log.gz 2023-10-11T11:23:25.463 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122069.log 2023-10-11T11:23:25.465 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.122069.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.122069.log.gz 2023-10-11T11:23:25.465 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119436.log 2023-10-11T11:23:25.467 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.119436.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.119436.log.gz 2023-10-11T11:23:25.467 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109845.log 2023-10-11T11:23:25.468 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.109845.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.109845.log.gz 2023-10-11T11:23:25.468 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.134196.log 2023-10-11T11:23:25.471 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.134196.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117422.log 2023-10-11T11:23:25.471 INFO:teuthology.orchestra.run.smithi111.stderr: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.134196.log.gz 2023-10-11T11:23:25.472 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.117422.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.117422.log.gz 2023-10-11T11:23:25.472 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118292.log 2023-10-11T11:23:25.473 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.118292.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.118292.log.gz 2023-10-11T11:23:25.473 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.133401.log 2023-10-11T11:23:25.475 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.133401.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.140353.log 2023-10-11T11:23:25.475 INFO:teuthology.orchestra.run.smithi111.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.133401.log.gz 2023-10-11T11:23:25.477 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.140353.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.140353.log.gz 2023-10-11T11:23:25.477 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124886.log 2023-10-11T11:23:25.478 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.124886.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.124886.log.gz 2023-10-11T11:23:25.478 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.127517.log 2023-10-11T11:23:25.479 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.127517.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.127517.log.gz 2023-10-11T11:23:25.479 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137603.log 2023-10-11T11:23:25.480 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.137603.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.137603.log.gz 2023-10-11T11:23:25.480 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.115483.log 2023-10-11T11:23:25.481 INFO:teuthology.orchestra.run.smithi050.stderr: 91.3% -- replaced with /var/log/ceph/cluster2-mgr.x.log.gz 2023-10-11T11:23:25.482 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.115483.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.115483.log.gz 2023-10-11T11:23:25.482 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.142950.log 2023-10-11T11:23:25.483 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.142950.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.142950.log.gz 2023-10-11T11:23:25.483 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.100557.log 2023-10-11T11:23:25.483 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.admin.100557.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.100557.log.gz 2023-10-11T11:23:25.483 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123077.log 2023-10-11T11:23:25.484 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.123077.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.123077.log.gz 2023-10-11T11:23:25.485 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.123842.log 2023-10-11T11:23:25.487 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.123842.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.133049.log 2023-10-11T11:23:25.487 INFO:teuthology.orchestra.run.smithi111.stderr: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.123842.log.gz 2023-10-11T11:23:25.488 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.133049.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.133049.log.gz 2023-10-11T11:23:25.488 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127448.log 2023-10-11T11:23:25.491 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.127448.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.127448.log.gz 2023-10-11T11:23:25.491 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.105239.log 2023-10-11T11:23:25.492 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.admin.105239.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.105239.log.gz 2023-10-11T11:23:25.493 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107262.log 2023-10-11T11:23:25.494 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.107262.log: 2.3% -- replaced with /var/log/ceph/cluster1-client.mirror.107262.log.gz 2023-10-11T11:23:25.494 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.134382.log 2023-10-11T11:23:25.496 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.134382.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.134382.log.gz 2023-10-11T11:23:25.496 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.111299.log 2023-10-11T11:23:25.497 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.admin.111299.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.admin.111299.log.gz 2023-10-11T11:23:25.497 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.134596.log 2023-10-11T11:23:25.498 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.134596.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.134596.log.gz 2023-10-11T11:23:25.498 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.146985.log 2023-10-11T11:23:25.499 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.admin.146985.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.146985.log.gz 2023-10-11T11:23:25.500 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127181.log 2023-10-11T11:23:25.500 INFO:teuthology.orchestra.run.smithi111.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.118154.log.gz 2023-10-11T11:23:25.501 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116457.log 2023-10-11T11:23:25.501 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.127181.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.127181.log.gz 2023-10-11T11:23:25.501 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.116457.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116008.log 2023-10-11T11:23:25.501 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.116008.log: 80.3% -- replaced with /var/log/ceph/cluster2-client.mirror.116008.log.gz 2023-10-11T11:23:25.502 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.122932.log 2023-10-11T11:23:25.504 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.122932.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.122932.log.gz 2023-10-11T11:23:25.504 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.127244.log 2023-10-11T11:23:25.505 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.127244.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.127244.log.gz 2023-10-11T11:23:25.505 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.135974.log 2023-10-11T11:23:25.507 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.135974.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.135974.log.gz 2023-10-11T11:23:25.507 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.129895.log 2023-10-11T11:23:25.508 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.129895.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.129895.log.gz 2023-10-11T11:23:25.508 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108663.log 2023-10-11T11:23:25.509 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.108663.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.108663.log.gz 2023-10-11T11:23:25.509 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.131861.log 2023-10-11T11:23:25.510 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.131861.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.131861.log.gz 2023-10-11T11:23:25.510 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119844.log 2023-10-11T11:23:25.512 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.119844.log: 80.3% -- replaced with /var/log/ceph/cluster2-client.mirror.119844.log.gz 2023-10-11T11:23:25.512 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.133657.log 2023-10-11T11:23:25.513 INFO:teuthology.orchestra.run.smithi111.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.115901.log.gz 2023-10-11T11:23:25.514 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.133069.log 2023-10-11T11:23:25.519 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.133069.log: /var/log/ceph/cluster2-client.mirror.133657.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.133657.log.gz 2023-10-11T11:23:25.519 INFO:teuthology.orchestra.run.smithi111.stderr: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.133069.log.gz 2023-10-11T11:23:25.519 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110405.log 2023-10-11T11:23:25.519 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.110405.log: 88.2% -- replaced with /var/log/ceph/cluster1-client.mirror.110405.log.gz 2023-10-11T11:23:25.520 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.131882.log 2023-10-11T11:23:25.520 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.131882.log: 92.3% -- replaced with /var/log/ceph/cluster2-client.mirror.131882.log.gz 2023-10-11T11:23:25.521 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.128679.log 2023-10-11T11:23:25.521 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121155.log 2023-10-11T11:23:25.521 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.128679.log: /var/log/ceph/cluster1-client.mirror.121155.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.128679.log.gz 2023-10-11T11:23:25.522 INFO:teuthology.orchestra.run.smithi111.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.121155.log.gz 2023-10-11T11:23:25.522 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.115503.log 2023-10-11T11:23:25.522 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.141512.log 2023-10-11T11:23:25.523 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.115503.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.115503.log.gz 2023-10-11T11:23:25.523 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.141512.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.141512.log.gz 2023-10-11T11:23:25.523 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.130518.log 2023-10-11T11:23:25.526 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.130518.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.130518.log.gz 2023-10-11T11:23:25.527 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137850.log 2023-10-11T11:23:25.527 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.137850.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.137850.log.gz 2023-10-11T11:23:25.528 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.129421.log 2023-10-11T11:23:25.530 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.129421.log: 92.0% -- replaced with /var/log/ceph/cluster1-client.mirror.129421.log.gz 2023-10-11T11:23:25.530 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.113564.log 2023-10-11T11:23:25.535 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.113564.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.113564.log.gz 2023-10-11T11:23:25.536 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.134238.log 2023-10-11T11:23:25.537 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.134238.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.134238.log.gz 2023-10-11T11:23:25.537 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122405.log 2023-10-11T11:23:25.538 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.122405.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.122405.log.gz 2023-10-11T11:23:25.538 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108607.log 2023-10-11T11:23:25.539 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.108607.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.108607.log.gz 2023-10-11T11:23:25.539 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.136036.log 2023-10-11T11:23:25.540 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.136036.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.136036.log.gz 2023-10-11T11:23:25.540 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109893.log 2023-10-11T11:23:25.541 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.109893.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.109893.log.gz 2023-10-11T11:23:25.542 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124313.log 2023-10-11T11:23:25.542 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.124313.log: 93.0% -- replaced with /var/log/ceph/cluster2-client.mirror.124313.log.gz 2023-10-11T11:23:25.543 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.130446.log 2023-10-11T11:23:25.543 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.130446.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.130446.log.gz 2023-10-11T11:23:25.543 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119419.log 2023-10-11T11:23:25.544 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.119419.log: 80.3% -- replaced with /var/log/ceph/cluster2-client.mirror.119419.log.gz 2023-10-11T11:23:25.544 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.111424.log 2023-10-11T11:23:25.544 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.111424.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.111424.log.gz 2023-10-11T11:23:25.545 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.143617.log 2023-10-11T11:23:25.545 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.admin.143617.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.143617.log.gz 2023-10-11T11:23:25.546 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121752.log 2023-10-11T11:23:25.546 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.121752.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.121752.log.gz 2023-10-11T11:23:25.547 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.114111.log 2023-10-11T11:23:25.552 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.114111.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.114111.log.gz 2023-10-11T11:23:25.552 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.141461.log 2023-10-11T11:23:25.553 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.141461.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.141461.log.gz 2023-10-11T11:23:25.553 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.132580.log 2023-10-11T11:23:25.555 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.132580.log: 92.1% -- replaced with /var/log/ceph/cluster1-client.mirror.132580.log.gz 2023-10-11T11:23:25.556 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.112651.log 2023-10-11T11:23:25.561 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.112651.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.112651.log.gz 2023-10-11T11:23:25.561 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.146847.log 2023-10-11T11:23:25.562 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.admin.146847.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.146847.log.gz 2023-10-11T11:23:25.562 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.142902.log 2023-10-11T11:23:25.563 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.142902.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.142902.log.gz 2023-10-11T11:23:25.564 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.106661.log 2023-10-11T11:23:25.564 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.106661.log: 74.3% -- replaced with /var/log/ceph/cluster2-client.mirror.106661.log.gz 2023-10-11T11:23:25.565 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111034.log 2023-10-11T11:23:25.570 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.111034.log: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.111034.log.gz 2023-10-11T11:23:25.570 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.120063.log 2023-10-11T11:23:25.581 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.120063.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.140611.log 2023-10-11T11:23:25.594 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.0.140611.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.100713.log 2023-10-11T11:23:25.595 INFO:teuthology.orchestra.run.smithi111.stderr: 93.3% -- replaced with /var/log/ceph/cluster2-client.mirror.0.140611.log.gz 2023-10-11T11:23:25.598 INFO:teuthology.orchestra.run.smithi111.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.109666.log.gz 2023-10-11T11:23:25.601 INFO:teuthology.orchestra.run.smithi050.stderr: 91.2% -- replaced with /var/log/ceph/cluster1-mgr.x.log.gz 2023-10-11T11:23:25.603 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.admin.100713.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.100713.log.gz 2023-10-11T11:23:25.604 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.117043.log 2023-10-11T11:23:25.604 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.128967.log 2023-10-11T11:23:25.604 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.117043.log: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.117043.log.gz 2023-10-11T11:23:25.605 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.128967.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.128967.log.gz 2023-10-11T11:23:25.606 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2.log 2023-10-11T11:23:25.612 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.127065.log 2023-10-11T11:23:25.612 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2.log: /var/log/ceph/cluster2-client.mirror.127065.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.127065.log.gz 2023-10-11T11:23:25.612 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122333.log 2023-10-11T11:23:25.612 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.122333.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.122333.log.gz 2023-10-11T11:23:25.613 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.129195.log 2023-10-11T11:23:25.613 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.129195.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.129195.log.gz 2023-10-11T11:23:25.613 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.129736.log 2023-10-11T11:23:25.613 INFO:teuthology.orchestra.run.smithi111.stderr: 88.1% -- replaced with /var/log/ceph/cluster2.log.gz 2023-10-11T11:23:25.614 INFO:teuthology.orchestra.run.smithi111.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.115413.log.gz 2023-10-11T11:23:25.614 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.129736.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.113626.log 2023-10-11T11:23:25.614 INFO:teuthology.orchestra.run.smithi111.stderr: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.129736.log.gz 2023-10-11T11:23:25.615 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.122541.log 2023-10-11T11:23:25.615 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.113626.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.121016.log 2023-10-11T11:23:25.615 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.122541.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.122541.log.gz 2023-10-11T11:23:25.616 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.121016.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.100367.log 2023-10-11T11:23:25.616 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.admin.100367.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.100367.log.gz 2023-10-11T11:23:25.616 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118095.log 2023-10-11T11:23:25.616 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.118095.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.121016.log.gz 2023-10-11T11:23:25.616 INFO:teuthology.orchestra.run.smithi111.stderr: 80.4% -- replaced with /var/log/ceph/cluster2-client.mirror.118095.log.gz 2023-10-11T11:23:25.617 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137805.log 2023-10-11T11:23:25.618 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.137805.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.128093.log 2023-10-11T11:23:25.618 INFO:teuthology.orchestra.run.smithi111.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.137805.log.gz 2023-10-11T11:23:25.619 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.128093.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.128093.log.gz 2023-10-11T11:23:25.619 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117481.log 2023-10-11T11:23:25.619 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.117481.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.117481.log.gz 2023-10-11T11:23:25.619 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.147077.log 2023-10-11T11:23:25.620 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.115583.log 2023-10-11T11:23:25.621 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.admin.147077.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.147077.log.gz 2023-10-11T11:23:25.621 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.107177.log 2023-10-11T11:23:25.621 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.115583.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.115583.log.gz 2023-10-11T11:23:25.621 INFO:teuthology.orchestra.run.smithi111.stderr: 92.8% -- replaced with /var/log/ceph/cluster2-client.mirror.113626.log.gz 2023-10-11T11:23:25.621 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.109602.log 2023-10-11T11:23:25.622 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.admin.107177.log: 79.8% -- replaced with /var/log/ceph/cluster1-client.admin.107177.log.gz 2023-10-11T11:23:25.626 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.109602.log: 93.3% -- replaced with /var/log/ceph/cluster2-client.mirror.109602.log.gz 2023-10-11T11:23:25.626 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.134831.log 2023-10-11T11:23:25.627 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123285.log 2023-10-11T11:23:25.627 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.134831.log: /var/log/ceph/cluster1-client.mirror.123285.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.123285.log.gz 2023-10-11T11:23:25.627 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.141720.log 2023-10-11T11:23:25.629 INFO:teuthology.orchestra.run.smithi111.stderr: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.134831.log.gz 2023-10-11T11:23:25.629 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.141720.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.141720.log.gz 2023-10-11T11:23:25.629 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124934.log 2023-10-11T11:23:25.630 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118776.log 2023-10-11T11:23:25.630 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.124934.log: /var/log/ceph/cluster2-client.mirror.118776.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.124934.log.gz 2023-10-11T11:23:25.640 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.120528.log 2023-10-11T11:23:25.642 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137693.log 2023-10-11T11:23:25.643 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.120528.log: /var/log/ceph/cluster1-client.mirror.137693.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.137693.log.gz 2023-10-11T11:23:25.643 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.146295.log 2023-10-11T11:23:25.643 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.admin.146295.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.120528.log.gz 2023-10-11T11:23:25.643 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.146295.log.gz 2023-10-11T11:23:25.643 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113326.log 2023-10-11T11:23:25.644 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.130833.log 2023-10-11T11:23:25.645 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.113326.log: /var/log/ceph/cluster1-client.mirror.130833.log: 92.1% -- replaced with /var/log/ceph/cluster1-client.mirror.130833.log.gz 2023-10-11T11:23:25.645 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109720.log 2023-10-11T11:23:25.646 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.109720.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.109720.log.gz 2023-10-11T11:23:25.646 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.132423.log 2023-10-11T11:23:25.648 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.132423.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.132423.log.gz 2023-10-11T11:23:25.648 INFO:teuthology.orchestra.run.smithi111.stderr: 93.1% -- replaced with /var/log/ceph/cluster1-client.mirror.113326.log.gz 2023-10-11T11:23:25.649 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.147261.log 2023-10-11T11:23:25.650 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.140332.log 2023-10-11T11:23:25.650 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.admin.147261.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.147261.log.gz 2023-10-11T11:23:25.650 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.142068.log 2023-10-11T11:23:25.650 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.140332.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.140332.log.gz 2023-10-11T11:23:25.650 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.142068.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120548.log 2023-10-11T11:23:25.651 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.120548.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.142068.log.gz 2023-10-11T11:23:25.651 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.121343.log 2023-10-11T11:23:25.652 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.121343.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.117830.log.gz 2023-10-11T11:23:25.652 INFO:teuthology.orchestra.run.smithi111.stderr: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.120548.log.gz 2023-10-11T11:23:25.652 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124732.log 2023-10-11T11:23:25.653 INFO:teuthology.orchestra.run.smithi111.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.121343.log.gz 2023-10-11T11:23:25.654 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.106866.log 2023-10-11T11:23:25.654 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.124732.log: /var/log/ceph/cluster1-client.mirror.106866.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.124732.log.gz 2023-10-11T11:23:25.654 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- 86.0%/var/log/ceph/cluster1-client.mirror.125567.log 2023-10-11T11:23:25.654 INFO:teuthology.orchestra.run.smithi111.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.106866.log.gz 2023-10-11T11:23:25.654 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.135767.log 2023-10-11T11:23:25.656 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.125567.log: /var/log/ceph/cluster2-client.mirror.135767.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.125567.log.gz 2023-10-11T11:23:25.656 INFO:teuthology.orchestra.run.smithi111.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.135767.log.gz 2023-10-11T11:23:25.656 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.130355.log 2023-10-11T11:23:25.656 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.128925.log 2023-10-11T11:23:25.657 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.130355.log: /var/log/ceph/cluster2-client.mirror.128925.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.130355.log.gz 2023-10-11T11:23:25.657 INFO:teuthology.orchestra.run.smithi111.stderr: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.128925.log.gz 2023-10-11T11:23:25.658 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123630.log 2023-10-11T11:23:25.658 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124819.log 2023-10-11T11:23:25.659 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.123630.log: /var/log/ceph/cluster2-client.mirror.124819.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.124819.log.gz 2023-10-11T11:23:25.659 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.112108.log 2023-10-11T11:23:25.660 INFO:teuthology.orchestra.run.smithi111.stderr: 92.1% -- replaced with /var/log/ceph/cluster1-client.mirror.123630.log.gz 2023-10-11T11:23:25.661 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.132874.log 2023-10-11T11:23:25.661 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.112108.log: /var/log/ceph/cluster1-client.mirror.132874.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.132874.log.gz 2023-10-11T11:23:25.661 INFO:teuthology.orchestra.run.smithi111.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.112108.log.gz 2023-10-11T11:23:25.662 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109941.log 2023-10-11T11:23:25.662 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117680.log 2023-10-11T11:23:25.663 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.109941.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.109941.log.gz 2023-10-11T11:23:25.663 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.121972.log 2023-10-11T11:23:25.663 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.117680.log: /var/log/ceph/cluster2-client.mirror.121972.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.117680.log.gz 2023-10-11T11:23:25.664 INFO:teuthology.orchestra.run.smithi111.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.121972.log.gz 2023-10-11T11:23:25.664 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.123420.log 2023-10-11T11:23:25.665 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126633.log 2023-10-11T11:23:25.665 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.123420.log: /var/log/ceph/cluster1-client.mirror.126633.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.126633.log.gz 2023-10-11T11:23:25.666 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.112224.log 2023-10-11T11:23:25.666 INFO:teuthology.orchestra.run.smithi111.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.123420.log.gz 2023-10-11T11:23:25.666 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.120220.log 2023-10-11T11:23:25.666 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.112224.log: /var/log/ceph/cluster2-client.mirror.120220.log: 1.3% -- replaced with /var/log/ceph/cluster2-client.mirror.120220.log.gz 2023-10-11T11:23:25.666 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110993.log 2023-10-11T11:23:25.667 INFO:teuthology.orchestra.run.smithi111.stderr: 91.2% -- replaced with /var/log/ceph/cluster2-client.mirror.112224.log.gz 2023-10-11T11:23:25.668 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117788.log 2023-10-11T11:23:25.668 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.110993.log: /var/log/ceph/cluster2-client.mirror.117788.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.110993.log.gz 2023-10-11T11:23:25.668 INFO:teuthology.orchestra.run.smithi111.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.117788.log.gz 2023-10-11T11:23:25.669 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124982.log 2023-10-11T11:23:25.669 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113260.log 2023-10-11T11:23:25.670 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.124982.log: /var/log/ceph/cluster1-client.mirror.113260.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.124982.log.gz 2023-10-11T11:23:25.670 INFO:teuthology.orchestra.run.smithi111.stderr: 89.4% -- replaced with /var/log/ceph/cluster1-client.mirror.113260.log.gz 2023-10-11T11:23:25.670 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.115209.log 2023-10-11T11:23:25.670 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.106544.log 2023-10-11T11:23:25.671 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.115209.log: /var/log/ceph/cluster2-client.admin.106544.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.106544.log.gz 2023-10-11T11:23:25.671 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117909.log 2023-10-11T11:23:25.682 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.117909.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137535.log 2023-10-11T11:23:25.687 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.137535.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.137535.log.gz 2023-10-11T11:23:25.695 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138098.log 2023-10-11T11:23:25.696 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.138098.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.138098.log.gz 2023-10-11T11:23:25.696 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.109746.log 2023-10-11T11:23:25.696 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.109746.log: 91.2% -- replaced with /var/log/ceph/cluster2-client.mirror.109746.log.gz 2023-10-11T11:23:25.697 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.145373.log 2023-10-11T11:23:25.697 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.admin.145373.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.145373.log.gz 2023-10-11T11:23:25.697 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.130994.log 2023-10-11T11:23:25.698 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.130994.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.130994.log.gz 2023-10-11T11:23:25.698 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.122426.log 2023-10-11T11:23:25.700 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.122426.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.122426.log.gz 2023-10-11T11:23:25.700 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.134811.log 2023-10-11T11:23:25.701 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.134811.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.134811.log.gz 2023-10-11T11:23:25.701 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.131781.log 2023-10-11T11:23:25.713 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.131781.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.131781.log.gz 2023-10-11T11:23:25.713 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.106832.log 2023-10-11T11:23:25.726 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.106832.log: 93.2% -- replaced with /var/log/ceph/cluster1-client.mirror.106832.log.gz 2023-10-11T11:23:25.727 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.134155.log 2023-10-11T11:23:25.728 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.134155.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.134155.log.gz 2023-10-11T11:23:25.728 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.131561.log 2023-10-11T11:23:25.729 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.131561.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.131561.log.gz 2023-10-11T11:23:25.729 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.2.106698.log 2023-10-11T11:23:25.735 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.2.106698.log: 93.8% -- replaced with /var/log/ceph/cluster1-client.mirror.2.106698.log.gz 2023-10-11T11:23:25.736 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.134320.log 2023-10-11T11:23:25.738 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.134320.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.134320.log.gz 2023-10-11T11:23:25.738 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138593.log 2023-10-11T11:23:25.739 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.138593.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.138593.log.gz 2023-10-11T11:23:25.739 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.141067.log 2023-10-11T11:23:25.740 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.141067.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.141067.log.gz 2023-10-11T11:23:25.740 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.110002.log 2023-10-11T11:23:25.745 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.0.110002.log: 93.6% -- replaced with /var/log/ceph/cluster1-client.mirror.0.110002.log.gz 2023-10-11T11:23:25.745 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.2.116774.log 2023-10-11T11:23:25.749 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.2.116774.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.131690.log 2023-10-11T11:23:25.771 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.131690.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.131690.log.gz 2023-10-11T11:23:25.771 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116596.log 2023-10-11T11:23:25.775 INFO:teuthology.orchestra.run.smithi111.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.118776.log.gz 2023-10-11T11:23:25.786 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.116596.log: 80.3% -- replaced with /var/log/ceph/cluster2-client.mirror.116596.log.gz 2023-10-11T11:23:25.786 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136883.log 2023-10-11T11:23:25.787 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.136883.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.136883.log.gz 2023-10-11T11:23:25.787 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.128384.log 2023-10-11T11:23:25.787 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.128384.log: 92.1% -- replaced with /var/log/ceph/cluster1-client.mirror.128384.log.gz 2023-10-11T11:23:25.787 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.145649.log 2023-10-11T11:23:25.787 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.admin.145649.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.145649.log.gz 2023-10-11T11:23:25.788 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.140863.log 2023-10-11T11:23:25.788 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.140863.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.120063.log.gz 2023-10-11T11:23:25.789 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.130672.log 2023-10-11T11:23:25.805 INFO:teuthology.orchestra.run.smithi111.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.119478.log.gz 2023-10-11T11:23:25.806 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.130672.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.130672.log.gz 2023-10-11T11:23:25.807 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137828.log 2023-10-11T11:23:25.807 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.137828.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.137828.log.gz 2023-10-11T11:23:25.808 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136523.log 2023-10-11T11:23:25.808 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.136523.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.136523.log.gz 2023-10-11T11:23:25.809 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.135705.log 2023-10-11T11:23:25.810 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.135705.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.135705.log.gz 2023-10-11T11:23:25.810 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.133422.log 2023-10-11T11:23:25.811 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.133422.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.133422.log.gz 2023-10-11T11:23:25.811 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.115958.log 2023-10-11T11:23:25.812 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.115958.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.115958.log.gz 2023-10-11T11:23:25.812 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.133574.log 2023-10-11T11:23:25.815 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.133574.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.133574.log.gz 2023-10-11T11:23:25.815 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.127311.log 2023-10-11T11:23:25.816 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.127311.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.127311.log.gz 2023-10-11T11:23:25.816 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.134114.log 2023-10-11T11:23:25.817 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.134114.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.134114.log.gz 2023-10-11T11:23:25.817 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117867.log 2023-10-11T11:23:25.818 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.117867.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.117867.log.gz 2023-10-11T11:23:25.818 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.127912.log 2023-10-11T11:23:25.820 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.127912.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.127912.log.gz 2023-10-11T11:23:25.820 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.114492.log 2023-10-11T11:23:25.822 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.114492.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.114492.log.gz 2023-10-11T11:23:25.822 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112374.log 2023-10-11T11:23:25.823 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.112374.log: 74.0% -- replaced with /var/log/ceph/cluster1-client.mirror.112374.log.gz 2023-10-11T11:23:25.823 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116655.log 2023-10-11T11:23:25.834 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.116655.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.105348.log 2023-10-11T11:23:25.839 INFO:teuthology.orchestra.run.smithi111.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.115209.log.gz 2023-10-11T11:23:25.840 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.admin.105348.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.105348.log.gz 2023-10-11T11:23:25.840 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.114883.log 2023-10-11T11:23:25.841 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.114883.log: 80.4% -- replaced with /var/log/ceph/cluster2-client.mirror.114883.log.gz 2023-10-11T11:23:25.841 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.134720.log 2023-10-11T11:23:25.843 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.134720.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.134720.log.gz 2023-10-11T11:23:25.843 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107313.log 2023-10-11T11:23:25.844 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.107313.log: 54.2% -- replaced with /var/log/ceph/cluster1-client.mirror.107313.log.gz 2023-10-11T11:23:25.844 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118191.log 2023-10-11T11:23:25.845 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.118191.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.118191.log.gz 2023-10-11T11:23:25.845 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.145281.log 2023-10-11T11:23:25.846 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.admin.145281.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.145281.log.gz 2023-10-11T11:23:25.846 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.128227.log 2023-10-11T11:23:25.847 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.128227.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.128227.log.gz 2023-10-11T11:23:25.847 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.131152.log 2023-10-11T11:23:25.848 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.131152.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.131152.log.gz 2023-10-11T11:23:25.848 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114131.log 2023-10-11T11:23:25.854 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.114131.log: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.114131.log.gz 2023-10-11T11:23:25.855 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.2.134933.log 2023-10-11T11:23:25.862 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.2.134933.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.2.134933.log.gz 2023-10-11T11:23:25.862 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.144859.log 2023-10-11T11:23:25.863 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.admin.144859.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109917.log 2023-10-11T11:23:25.863 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.144859.log.gz 2023-10-11T11:23:25.863 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.109917.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.109917.log.gz 2023-10-11T11:23:25.863 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.111616.log 2023-10-11T11:23:25.869 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.111616.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.111616.log.gz 2023-10-11T11:23:25.869 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.133616.log 2023-10-11T11:23:25.871 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.133616.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.133616.log.gz 2023-10-11T11:23:25.871 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.113626.log 2023-10-11T11:23:25.872 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.114532.log 2023-10-11T11:23:25.872 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.114532.log: /var/log/ceph/cluster1-client.mirror.0.113626.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.114532.log.gz 2023-10-11T11:23:25.872 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.133173.log 2023-10-11T11:23:25.873 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.133173.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.133173.log.gz 2023-10-11T11:23:25.873 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127407.log 2023-10-11T11:23:25.874 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.127407.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.127407.log.gz 2023-10-11T11:23:25.875 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.130292.log 2023-10-11T11:23:25.875 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.130292.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.130292.log.gz 2023-10-11T11:23:25.876 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.106268.log 2023-10-11T11:23:25.885 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.admin.106268.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.106268.log.gz 2023-10-11T11:23:25.886 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.119288.log 2023-10-11T11:23:25.893 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.0.119288.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.0.119288.log.gz 2023-10-11T11:23:25.893 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108639.log 2023-10-11T11:23:25.894 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.108639.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.108639.log.gz 2023-10-11T11:23:25.894 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118272.log 2023-10-11T11:23:25.896 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.118272.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.118272.log.gz 2023-10-11T11:23:25.896 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109622.log 2023-10-11T11:23:25.898 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.109622.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124061.log 2023-10-11T11:23:25.902 INFO:teuthology.orchestra.run.smithi111.stderr: 93.2% -- replaced with /var/log/ceph/cluster1-client.mirror.109622.log.gz 2023-10-11T11:23:25.903 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.124061.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.124061.log.gz 2023-10-11T11:23:25.903 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.129082.log 2023-10-11T11:23:25.904 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.129082.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.129082.log.gz 2023-10-11T11:23:25.904 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117402.log 2023-10-11T11:23:25.906 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.117402.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.117402.log.gz 2023-10-11T11:23:25.906 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.129443.log 2023-10-11T11:23:25.907 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.129443.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.129443.log.gz 2023-10-11T11:23:25.908 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.109398.log 2023-10-11T11:23:25.909 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.109398.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.109398.log.gz 2023-10-11T11:23:25.910 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117974.log 2023-10-11T11:23:25.911 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.117974.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.117974.log.gz 2023-10-11T11:23:25.911 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.145557.log 2023-10-11T11:23:25.911 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.admin.145557.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.145557.log.gz 2023-10-11T11:23:25.912 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119570.log 2023-10-11T11:23:25.916 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.119570.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.123652.log 2023-10-11T11:23:25.920 INFO:teuthology.orchestra.run.smithi111.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.116457.log.gz 2023-10-11T11:23:25.926 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.123652.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.117909.log.gz 2023-10-11T11:23:25.926 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118174.log 2023-10-11T11:23:25.926 INFO:teuthology.orchestra.run.smithi111.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.123652.log.gz 2023-10-11T11:23:25.927 INFO:teuthology.orchestra.run.smithi111.stderr: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.0.113626.log.gz 2023-10-11T11:23:25.927 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.118174.log: 80.2% -- replaced with /var/log/ceph/cluster2-client.mirror.118174.log.gz 2023-10-11T11:23:25.927 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.131668.log 2023-10-11T11:23:25.927 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137940.log 2023-10-11T11:23:25.927 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.131668.log: /var/log/ceph/cluster1-client.mirror.137940.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.137940.log.gz 2023-10-11T11:23:25.928 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.130542.log 2023-10-11T11:23:25.928 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.130542.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.130542.log.gz 2023-10-11T11:23:25.928 INFO:teuthology.orchestra.run.smithi111.stderr: 92.0% -- replaced with /var/log/ceph/cluster1-client.mirror.131668.log.gz 2023-10-11T11:23:25.928 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.142319.log 2023-10-11T11:23:25.928 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113498.log 2023-10-11T11:23:25.929 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.142319.log: /var/log/ceph/cluster1-client.mirror.113498.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.142319.log.gz 2023-10-11T11:23:25.931 INFO:teuthology.orchestra.run.smithi111.stderr: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.113498.log.gz 2023-10-11T11:23:25.932 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.132087.log 2023-10-11T11:23:25.932 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116568.log 2023-10-11T11:23:25.933 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.132087.log: /var/log/ceph/cluster2-client.mirror.116568.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.132087.log.gz 2023-10-11T11:23:25.933 INFO:teuthology.orchestra.run.smithi111.stderr: 93.0% -- replaced with /var/log/ceph/cluster1-client.mirror.2.116774.log.gz 2023-10-11T11:23:25.934 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107279.log 2023-10-11T11:23:25.934 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.142154.log 2023-10-11T11:23:25.934 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.107279.log: 57.4% -- replaced with /var/log/ceph/cluster1-client.mirror.107279.log.gz 2023-10-11T11:23:25.935 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.142154.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.142154.log.gz 2023-10-11T11:23:25.935 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109208.log 2023-10-11T11:23:25.936 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138480.log 2023-10-11T11:23:25.936 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.109208.log: /var/log/ceph/cluster1-client.mirror.138480.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.138480.log.gz 2023-10-11T11:23:25.936 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116361.log 2023-10-11T11:23:25.940 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.116361.log: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.109208.log.gz 2023-10-11T11:23:25.940 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.130244.log 2023-10-11T11:23:25.941 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.130244.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.130244.log.gz 2023-10-11T11:23:25.942 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113779.log 2023-10-11T11:23:25.945 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.135829.log 2023-10-11T11:23:25.947 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.113779.log: /var/log/ceph/cluster2-client.mirror.135829.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.135829.log.gz 2023-10-11T11:23:25.947 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116712.log 2023-10-11T11:23:25.948 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.116712.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.116712.log.gz 2023-10-11T11:23:25.948 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122845.log 2023-10-11T11:23:25.949 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.122845.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.122845.log.gz 2023-10-11T11:23:25.949 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.135085.log 2023-10-11T11:23:25.951 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.135085.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.135085.log.gz 2023-10-11T11:23:25.951 INFO:teuthology.orchestra.run.smithi111.stderr: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.113779.log.gz 2023-10-11T11:23:25.951 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.134741.log 2023-10-11T11:23:25.952 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.134933.log 2023-10-11T11:23:25.952 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.134741.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.134741.log.gz 2023-10-11T11:23:25.952 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.134933.log: 90.8% -- replaced with /var/log/ceph/cluster1-client.mirror.134933.log.gz 2023-10-11T11:23:25.952 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125382.log 2023-10-11T11:23:25.953 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.125382.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.105853.log 2023-10-11T11:23:25.958 INFO:teuthology.orchestra.run.smithi111.stderr: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.125382.log.gz 2023-10-11T11:23:25.958 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.admin.105853.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.105853.log.gz 2023-10-11T11:23:25.958 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.113759.log 2023-10-11T11:23:25.958 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.113759.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.135436.log 2023-10-11T11:23:25.960 INFO:teuthology.orchestra.run.smithi111.stderr: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.113759.log.gz 2023-10-11T11:23:25.963 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.135436.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.135436.log.gz 2023-10-11T11:23:25.963 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125793.log 2023-10-11T11:23:25.964 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.125793.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.125793.log.gz 2023-10-11T11:23:25.964 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.146939.log 2023-10-11T11:23:25.965 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.admin.146939.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.146939.log.gz 2023-10-11T11:23:25.965 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138120.log 2023-10-11T11:23:25.966 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.138120.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.138120.log.gz 2023-10-11T11:23:25.966 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.135147.log 2023-10-11T11:23:25.968 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.135147.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.135147.log.gz 2023-10-11T11:23:25.968 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.144629.log 2023-10-11T11:23:25.969 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.admin.144629.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.144629.log.gz 2023-10-11T11:23:25.969 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.2.108868.log 2023-10-11T11:23:25.975 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.2.108868.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.141683.log 2023-10-11T11:23:25.976 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.141683.log: 80.6% -- replaced with /var/log/ceph/cluster2-client.mirror.141683.log.gz 2023-10-11T11:23:25.977 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.129784.log 2023-10-11T11:23:25.978 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.129784.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.129784.log.gz 2023-10-11T11:23:25.978 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126907.log 2023-10-11T11:23:25.982 INFO:teuthology.orchestra.run.smithi111.stderr: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.2.108868.log.gz 2023-10-11T11:23:25.982 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120241.log 2023-10-11T11:23:25.982 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.126907.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.126907.log.gz 2023-10-11T11:23:25.983 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.120241.log: 2.6% -- replaced with /var/log/ceph/cluster1-client.mirror.120241.log.gz 2023-10-11T11:23:25.983 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.111573.log 2023-10-11T11:23:25.988 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.111573.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.126835.log 2023-10-11T11:23:25.989 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.126835.log: 93.3% -- replaced with /var/log/ceph/cluster2-client.mirror.111573.log.gz 2023-10-11T11:23:25.989 INFO:teuthology.orchestra.run.smithi111.stderr: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.126835.log.gz 2023-10-11T11:23:25.989 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.141016.log 2023-10-11T11:23:25.990 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.141016.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.106360.log 2023-10-11T11:23:25.990 INFO:teuthology.orchestra.run.smithi111.stderr: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.141016.log.gz 2023-10-11T11:23:25.991 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.admin.106360.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.106360.log.gz 2023-10-11T11:23:25.991 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.111057.log 2023-10-11T11:23:25.991 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.111057.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.142687.log 2023-10-11T11:23:25.992 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.142687.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.142687.log.gz 2023-10-11T11:23:25.993 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.128343.log 2023-10-11T11:23:25.994 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.128343.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.128343.log.gz 2023-10-11T11:23:25.994 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.146893.log 2023-10-11T11:23:25.994 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.admin.146893.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.146893.log.gz 2023-10-11T11:23:25.995 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118593.log 2023-10-11T11:23:26.005 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.118593.log: 93.3% -- replaced with /var/log/ceph/cluster2-client.mirror.111057.log.gz 2023-10-11T11:23:26.006 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125245.log 2023-10-11T11:23:26.006 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.125245.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.125245.log.gz 2023-10-11T11:23:26.007 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.133442.log 2023-10-11T11:23:26.007 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.133442.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.133442.log.gz 2023-10-11T11:23:26.007 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.125903.log 2023-10-11T11:23:26.007 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.125903.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.125903.log.gz 2023-10-11T11:23:26.008 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120667.log 2023-10-11T11:23:26.008 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.120667.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.120667.log.gz 2023-10-11T11:23:26.008 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.131063.log 2023-10-11T11:23:26.017 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.131063.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.131063.log.gz 2023-10-11T11:23:26.018 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.132620.log 2023-10-11T11:23:26.018 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.132620.log: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.132620.log.gz 2023-10-11T11:23:26.018 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.129034.log 2023-10-11T11:23:26.018 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.129034.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.129034.log.gz 2023-10-11T11:23:26.018 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.135312.log 2023-10-11T11:23:26.019 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.135312.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.135312.log.gz 2023-10-11T11:23:26.019 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138390.log 2023-10-11T11:23:26.019 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.138390.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.138390.log.gz 2023-10-11T11:23:26.019 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.107585.log 2023-10-11T11:23:26.019 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.107585.log: 80.4% -- replaced with /var/log/ceph/cluster2-client.mirror.107585.log.gz 2023-10-11T11:23:26.020 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126205.log 2023-10-11T11:23:26.020 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.126205.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.126205.log.gz 2023-10-11T11:23:26.020 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122117.log 2023-10-11T11:23:26.020 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.122117.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.122117.log.gz 2023-10-11T11:23:26.020 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116880.log 2023-10-11T11:23:26.020 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.116880.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.116880.log.gz 2023-10-11T11:23:26.021 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.108984.log 2023-10-11T11:23:26.021 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.108984.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.108984.log.gz 2023-10-11T11:23:26.022 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.105394.log 2023-10-11T11:23:26.022 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.admin.105394.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.105394.log.gz 2023-10-11T11:23:26.022 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136703.log 2023-10-11T11:23:26.022 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.136703.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.136703.log.gz 2023-10-11T11:23:26.022 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111448.log 2023-10-11T11:23:26.023 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.111448.log: 88.6% -- replaced with /var/log/ceph/cluster1-client.mirror.111448.log.gz 2023-10-11T11:23:26.023 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120459.log 2023-10-11T11:23:26.024 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.120459.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.120459.log.gz 2023-10-11T11:23:26.024 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111528.log 2023-10-11T11:23:26.024 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.111528.log: 88.6% -- replaced with /var/log/ceph/cluster1-client.mirror.111528.log.gz 2023-10-11T11:23:26.025 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-osd.2.log 2023-10-11T11:23:26.028 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-osd.2.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.147123.log 2023-10-11T11:23:26.047 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.admin.147123.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.147123.log.gz 2023-10-11T11:23:26.047 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127222.log 2023-10-11T11:23:26.058 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.127222.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.135354.log 2023-10-11T11:23:26.058 INFO:teuthology.orchestra.run.smithi111.stderr: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.127222.log.gz 2023-10-11T11:23:26.076 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.135354.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.135354.log.gz 2023-10-11T11:23:26.076 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.127202.log 2023-10-11T11:23:26.076 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.127202.log: gzip -5 --verbose -- /var/log/ceph/cluster2-osd.1.log 2023-10-11T11:23:26.086 INFO:teuthology.orchestra.run.smithi111.stderr: 92.3% -- replaced with /var/log/ceph/cluster2-client.mirror.127202.log.gz 2023-10-11T11:23:26.095 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-osd.1.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.116655.log.gz 2023-10-11T11:23:26.099 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124551.log 2023-10-11T11:23:26.118 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.124551.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.124551.log.gz 2023-10-11T11:23:26.119 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126520.log 2023-10-11T11:23:26.130 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.126520.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.126520.log.gz 2023-10-11T11:23:26.130 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.132919.log 2023-10-11T11:23:26.152 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.132919.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.132919.log.gz 2023-10-11T11:23:26.152 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117219.log 2023-10-11T11:23:26.163 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.117219.log: 80.3% -- replaced with /var/log/ceph/cluster2-client.mirror.117219.log.gz 2023-10-11T11:23:26.163 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138053.log 2023-10-11T11:23:26.174 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.138053.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.138053.log.gz 2023-10-11T11:23:26.174 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.140609.log 2023-10-11T11:23:26.185 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.140609.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.140609.log.gz 2023-10-11T11:23:26.185 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.115284.log 2023-10-11T11:23:26.185 INFO:teuthology.orchestra.run.smithi111.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.116361.log.gz 2023-10-11T11:23:26.187 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.115284.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.115284.log.gz 2023-10-11T11:23:26.188 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.141786.log 2023-10-11T11:23:26.188 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.141786.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.141786.log.gz 2023-10-11T11:23:26.188 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.131757.log 2023-10-11T11:23:26.189 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.131757.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.131757.log.gz 2023-10-11T11:23:26.189 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.129014.log 2023-10-11T11:23:26.191 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.129014.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.129014.log.gz 2023-10-11T11:23:26.191 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.135478.log 2023-10-11T11:23:26.192 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.135478.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.135478.log.gz 2023-10-11T11:23:26.192 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.121704.log 2023-10-11T11:23:26.193 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.121704.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.121704.log.gz 2023-10-11T11:23:26.193 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.142390.log 2023-10-11T11:23:26.194 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.142390.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.142390.log.gz 2023-10-11T11:23:26.194 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.135106.log 2023-10-11T11:23:26.195 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.135106.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.135106.log.gz 2023-10-11T11:23:26.195 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123981.log 2023-10-11T11:23:26.196 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.123981.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.123981.log.gz 2023-10-11T11:23:26.197 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113123.log 2023-10-11T11:23:26.202 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.113123.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.113123.log.gz 2023-10-11T11:23:26.203 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.135292.log 2023-10-11T11:23:26.204 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.135292.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.135292.log.gz 2023-10-11T11:23:26.204 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.2.136096.log 2023-10-11T11:23:26.206 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.2.136096.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.131220.log 2023-10-11T11:23:26.211 INFO:teuthology.orchestra.run.smithi111.stderr: 93.8% -- replaced with /var/log/ceph/cluster1-client.mirror.2.136096.log.gz 2023-10-11T11:23:26.211 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.131220.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.131220.log.gz 2023-10-11T11:23:26.212 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107228.log 2023-10-11T11:23:26.212 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.107228.log: 54.4% -- replaced with /var/log/ceph/cluster1-client.mirror.107228.log.gz 2023-10-11T11:23:26.213 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.112490.log 2023-10-11T11:23:26.214 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.112490.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.112490.log.gz 2023-10-11T11:23:26.214 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117887.log 2023-10-11T11:23:26.215 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.117887.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.117887.log.gz 2023-10-11T11:23:26.216 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.143571.log 2023-10-11T11:23:26.216 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.admin.143571.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.143571.log.gz 2023-10-11T11:23:26.217 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116240.log 2023-10-11T11:23:26.218 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.116240.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.116240.log.gz 2023-10-11T11:23:26.218 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.115267.log 2023-10-11T11:23:26.219 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.115267.log: 80.5% -- replaced with /var/log/ceph/cluster2-client.mirror.115267.log.gz 2023-10-11T11:23:26.219 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.115371.log 2023-10-11T11:23:26.221 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.115371.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.115371.log.gz 2023-10-11T11:23:26.221 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.130107.log 2023-10-11T11:23:26.223 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.130107.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.130107.log.gz 2023-10-11T11:23:26.224 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138458.log 2023-10-11T11:23:26.224 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.138458.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.138458.log.gz 2023-10-11T11:23:26.224 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.2.124313.log 2023-10-11T11:23:26.227 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.2.124313.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.107622.log 2023-10-11T11:23:26.233 INFO:teuthology.orchestra.run.smithi111.stderr: 93.6% -- replaced with /var/log/ceph/cluster1-client.mirror.2.124313.log.gz 2023-10-11T11:23:26.233 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.107622.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.107622.log.gz 2023-10-11T11:23:26.233 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.141362.log 2023-10-11T11:23:26.234 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.141362.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.141362.log.gz 2023-10-11T11:23:26.234 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136323.log 2023-10-11T11:23:26.235 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.136323.log: 3.9% -- replaced with /var/log/ceph/cluster1-client.mirror.136323.log.gz 2023-10-11T11:23:26.235 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.132539.log 2023-10-11T11:23:26.236 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.132539.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.132539.log.gz 2023-10-11T11:23:26.236 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.115704.log 2023-10-11T11:23:26.238 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.115704.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.128815.log 2023-10-11T11:23:26.247 INFO:teuthology.orchestra.run.smithi111.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.140863.log.gz 2023-10-11T11:23:26.250 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.128815.log: 73.7% -- replaced with /var/log/ceph/cluster1-client.mirror.128815.log.gz 2023-10-11T11:23:26.251 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.123005.log 2023-10-11T11:23:26.252 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.123005.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.123005.log.gz 2023-10-11T11:23:26.252 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136455.log 2023-10-11T11:23:26.253 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.136455.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.136455.log.gz 2023-10-11T11:23:26.253 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.139887.log 2023-10-11T11:23:26.256 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.139887.log: 91.2% -- replaced with /var/log/ceph/cluster2-client.mirror.139887.log.gz 2023-10-11T11:23:26.256 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.143847.log 2023-10-11T11:23:26.256 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.admin.143847.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.143847.log.gz 2023-10-11T11:23:26.257 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.132128.log 2023-10-11T11:23:26.259 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.132128.log: 92.1% -- replaced with /var/log/ceph/cluster1-client.mirror.132128.log.gz 2023-10-11T11:23:26.259 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108559.log 2023-10-11T11:23:26.260 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.108559.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.108559.log.gz 2023-10-11T11:23:26.260 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138368.log 2023-10-11T11:23:26.261 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.138368.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.138368.log.gz 2023-10-11T11:23:26.261 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117365.log 2023-10-11T11:23:26.272 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.117365.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116546.log 2023-10-11T11:23:26.292 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.116546.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.116546.log.gz 2023-10-11T11:23:26.292 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109579.log 2023-10-11T11:23:26.299 INFO:teuthology.orchestra.run.smithi050.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-mon.a.log.gz 2023-10-11T11:23:26.303 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.109579.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126657.log 2023-10-11T11:23:26.304 INFO:teuthology.orchestra.run.smithi111.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.118593.log.gz 2023-10-11T11:23:26.304 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.126657.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.126657.log.gz 2023-10-11T11:23:26.304 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.140742.log 2023-10-11T11:23:26.305 INFO:teuthology.orchestra.run.smithi111.stderr: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.109579.log.gz 2023-10-11T11:23:26.305 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.140742.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.140742.log.gz 2023-10-11T11:23:26.305 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.136277.log 2023-10-11T11:23:26.305 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.136277.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.136277.log.gz 2023-10-11T11:23:26.305 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.140219.log 2023-10-11T11:23:26.306 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.140219.log: 81.3% -- replaced with /var/log/ceph/cluster2-client.mirror.140219.log.gz 2023-10-11T11:23:26.315 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124866.log 2023-10-11T11:23:26.315 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.124866.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.124866.log.gz 2023-10-11T11:23:26.315 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.3.116107.log 2023-10-11T11:23:26.315 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.3.116107.log: 92.7% -- replaced with /var/log/ceph/cluster2-client.mirror.3.116107.log.gz 2023-10-11T11:23:26.316 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.115391.log 2023-10-11T11:23:26.316 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.115391.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.115391.log.gz 2023-10-11T11:23:26.316 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136500.log 2023-10-11T11:23:26.316 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.136500.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.136500.log.gz 2023-10-11T11:23:26.316 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.141318.log 2023-10-11T11:23:26.317 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.141318.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.141318.log.gz 2023-10-11T11:23:26.317 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109052.log 2023-10-11T11:23:26.317 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.109052.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.109052.log.gz 2023-10-11T11:23:26.317 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.112023.log 2023-10-11T11:23:26.330 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.112023.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.112023.log.gz 2023-10-11T11:23:26.330 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.136186.log 2023-10-11T11:23:26.330 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.136186.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.136186.log.gz 2023-10-11T11:23:26.331 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109490.log 2023-10-11T11:23:26.331 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.109490.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.109490.log.gz 2023-10-11T11:23:26.331 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116613.log 2023-10-11T11:23:26.331 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.116613.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.116613.log.gz 2023-10-11T11:23:26.332 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118075.log 2023-10-11T11:23:26.341 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.118075.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.130768.log 2023-10-11T11:23:26.341 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.130768.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.130768.log.gz 2023-10-11T11:23:26.341 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.134010.log 2023-10-11T11:23:26.355 INFO:teuthology.orchestra.run.smithi111.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.119570.log.gz 2023-10-11T11:23:26.362 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.134010.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.109644.log 2023-10-11T11:23:26.363 INFO:teuthology.orchestra.run.smithi111.stderr: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.134010.log.gz 2023-10-11T11:23:26.363 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.109644.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.109644.log.gz 2023-10-11T11:23:26.370 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.133525.log 2023-10-11T11:23:26.381 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.133525.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119793.log 2023-10-11T11:23:26.382 INFO:teuthology.orchestra.run.smithi111.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.133525.log.gz 2023-10-11T11:23:26.387 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.119793.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.119793.log.gz 2023-10-11T11:23:26.387 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.144215.log 2023-10-11T11:23:26.388 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.admin.144215.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.144215.log.gz 2023-10-11T11:23:26.388 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.133947.log 2023-10-11T11:23:26.391 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.133947.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.133947.log.gz 2023-10-11T11:23:26.391 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123029.log 2023-10-11T11:23:26.392 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.123029.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.123029.log.gz 2023-10-11T11:23:26.392 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109538.log 2023-10-11T11:23:26.393 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.109538.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.109538.log.gz 2023-10-11T11:23:26.393 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126027.log 2023-10-11T11:23:26.394 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.126027.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.126027.log.gz 2023-10-11T11:23:26.395 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.115304.log 2023-10-11T11:23:26.395 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.115304.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.115304.log.gz 2023-10-11T11:23:26.396 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.143525.log 2023-10-11T11:23:26.396 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.admin.143525.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.143525.log.gz 2023-10-11T11:23:26.396 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.114512.log 2023-10-11T11:23:26.398 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.114512.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116435.log 2023-10-11T11:23:26.398 INFO:teuthology.orchestra.run.smithi111.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.114512.log.gz 2023-10-11T11:23:26.399 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.116435.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.116435.log.gz 2023-10-11T11:23:26.399 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.139760.log 2023-10-11T11:23:26.400 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.139760.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.139760.log.gz 2023-10-11T11:23:26.401 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.141742.log 2023-10-11T11:23:26.412 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.141742.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.133864.log 2023-10-11T11:23:26.431 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.133864.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.133864.log.gz 2023-10-11T11:23:26.431 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.120103.log 2023-10-11T11:23:26.442 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.120103.log: 91.7% -- replaced with /var/log/ceph/cluster2-client.mirror.120103.log.gz 2023-10-11T11:23:26.443 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136860.log 2023-10-11T11:23:26.443 INFO:teuthology.orchestra.run.smithi111.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.116568.log.gz 2023-10-11T11:23:26.443 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.136860.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.136860.log.gz 2023-10-11T11:23:26.443 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124269.log 2023-10-11T11:23:26.443 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.124269.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.124269.log.gz 2023-10-11T11:23:26.444 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.106680.log 2023-10-11T11:23:26.452 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.1.106680.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.128247.log 2023-10-11T11:23:26.452 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.128247.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.128247.log.gz 2023-10-11T11:23:26.452 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.145511.log 2023-10-11T11:23:26.473 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.admin.145511.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.145511.log.gz 2023-10-11T11:23:26.473 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.120021.log 2023-10-11T11:23:26.496 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.120021.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.120021.log.gz 2023-10-11T11:23:26.496 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116045.log 2023-10-11T11:23:26.507 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.116045.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.116045.log.gz 2023-10-11T11:23:26.507 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.114068.log 2023-10-11T11:23:26.518 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.114068.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.135064.log 2023-10-11T11:23:26.518 INFO:teuthology.orchestra.run.smithi111.stderr: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.114068.log.gz 2023-10-11T11:23:26.530 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.135064.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138413.log 2023-10-11T11:23:26.541 INFO:teuthology.orchestra.run.smithi111.stderr: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.135064.log.gz 2023-10-11T11:23:26.552 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.138413.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.138413.log.gz 2023-10-11T11:23:26.552 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.109801.log 2023-10-11T11:23:26.562 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.109801.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.109801.log.gz 2023-10-11T11:23:26.563 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.133236.log 2023-10-11T11:23:26.580 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.133236.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.133236.log.gz 2023-10-11T11:23:26.580 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.114793.log 2023-10-11T11:23:26.594 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.114793.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.114793.log.gz 2023-10-11T11:23:26.594 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110646.log 2023-10-11T11:23:26.608 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.110646.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.114197.log 2023-10-11T11:23:26.618 INFO:teuthology.orchestra.run.smithi111.stderr: 93.9% -- replaced with /var/log/ceph/cluster1-client.mirror.1.106680.log.gz 2023-10-11T11:23:26.618 INFO:teuthology.orchestra.run.smithi111.stderr: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.110646.log.gz 2023-10-11T11:23:26.619 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.114197.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.114197.log.gz 2023-10-11T11:23:26.619 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.107211.log 2023-10-11T11:23:26.619 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.admin.107211.log: 67.2% -- replaced with /var/log/ceph/cluster1-client.admin.107211.log.gz 2023-10-11T11:23:26.620 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.122289.log 2023-10-11T11:23:26.622 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.122289.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.122289.log.gz 2023-10-11T11:23:26.623 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.108535.log 2023-10-11T11:23:26.623 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.108535.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.108535.log.gz 2023-10-11T11:23:26.623 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.140474.log 2023-10-11T11:23:26.628 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.140474.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127359.log 2023-10-11T11:23:26.641 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.127359.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.127359.log.gz 2023-10-11T11:23:26.641 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.128945.log 2023-10-11T11:23:26.648 INFO:teuthology.orchestra.run.smithi111.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.115704.log.gz 2023-10-11T11:23:26.654 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.128945.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.128945.log.gz 2023-10-11T11:23:26.655 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136590.log 2023-10-11T11:23:26.657 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.136590.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.136590.log.gz 2023-10-11T11:23:26.658 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.108828.log 2023-10-11T11:23:26.659 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.108828.log: 91.5% -- replaced with /var/log/ceph/cluster2-client.mirror.108828.log.gz 2023-10-11T11:23:26.660 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123053.log 2023-10-11T11:23:26.660 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.123053.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.123053.log.gz 2023-10-11T11:23:26.661 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.121290.log 2023-10-11T11:23:26.662 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.121290.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.121290.log.gz 2023-10-11T11:23:26.662 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.126976.log 2023-10-11T11:23:26.664 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.126976.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.117365.log.gz 2023-10-11T11:23:26.664 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.109295.log 2023-10-11T11:23:26.665 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.admin.109295.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.126976.log.gz 2023-10-11T11:23:26.665 INFO:teuthology.orchestra.run.smithi111.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.109295.log.gz 2023-10-11T11:23:26.666 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120715.log 2023-10-11T11:23:26.667 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.125471.log 2023-10-11T11:23:26.668 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.120715.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.120715.log.gz 2023-10-11T11:23:26.668 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.125471.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123333.log 2023-10-11T11:23:26.668 INFO:teuthology.orchestra.run.smithi111.stderr: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.125471.log.gz 2023-10-11T11:23:26.668 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.123333.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.139846.log 2023-10-11T11:23:26.668 INFO:teuthology.orchestra.run.smithi111.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.123333.log.gz 2023-10-11T11:23:26.669 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.139846.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.113521.log 2023-10-11T11:23:26.669 INFO:teuthology.orchestra.run.smithi111.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.139846.log.gz 2023-10-11T11:23:26.669 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.113521.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.144169.log 2023-10-11T11:23:26.670 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.admin.144169.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.144169.log.gz 2023-10-11T11:23:26.670 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.105992.log 2023-10-11T11:23:26.671 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.admin.105992.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.105992.log.gz 2023-10-11T11:23:26.671 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.146479.log 2023-10-11T11:23:26.671 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.admin.146479.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.146479.log.gz 2023-10-11T11:23:26.671 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.115566.log 2023-10-11T11:23:26.672 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.115566.log: 80.5% -- replaced with /var/log/ceph/cluster2-client.mirror.115566.log.gz 2023-10-11T11:23:26.672 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.105302.log 2023-10-11T11:23:26.672 INFO:teuthology.orchestra.run.smithi111.stderr: 93.3% -- replaced with /var/log/ceph/cluster2-client.mirror.113521.log.gz 2023-10-11T11:23:26.673 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.admin.105302.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.105302.log.gz 2023-10-11T11:23:26.674 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.140704.log 2023-10-11T11:23:26.674 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.140704.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116692.log 2023-10-11T11:23:26.674 INFO:teuthology.orchestra.run.smithi111.stderr: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.140704.log.gz 2023-10-11T11:23:26.675 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.116692.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.116692.log.gz 2023-10-11T11:23:26.675 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118393.log 2023-10-11T11:23:26.676 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126681.log 2023-10-11T11:23:26.676 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.118393.log: /var/log/ceph/cluster1-client.mirror.126681.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.126681.log.gz 2023-10-11T11:23:26.677 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.108308.log 2023-10-11T11:23:26.678 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.admin.108308.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.108308.log.gz 2023-10-11T11:23:26.678 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.115150.log 2023-10-11T11:23:26.678 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.115150.log: 80.3% -- replaced with /var/log/ceph/cluster2-client.mirror.115150.log.gz 2023-10-11T11:23:26.679 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124958.log 2023-10-11T11:23:26.680 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.124958.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.124958.log.gz 2023-10-11T11:23:26.680 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.127291.log 2023-10-11T11:23:26.681 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.127291.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.127291.log.gz 2023-10-11T11:23:26.682 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.119014.log 2023-10-11T11:23:26.687 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.119014.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.118075.log.gz 2023-10-11T11:23:26.688 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124708.log 2023-10-11T11:23:26.688 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.124708.log: 95.0% -- replaced with /var/log/ceph/cluster1-client.mirror.119014.log.gz 2023-10-11T11:23:26.688 INFO:teuthology.orchestra.run.smithi111.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.124708.log.gz 2023-10-11T11:23:26.688 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.138652.log 2023-10-11T11:23:26.689 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.138652.log: 80.5% -- replaced with /var/log/ceph/cluster2-client.mirror.138652.log.gz 2023-10-11T11:23:26.689 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.129647.log 2023-10-11T11:23:26.690 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.129647.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.129647.log.gz 2023-10-11T11:23:26.691 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.135870.log 2023-10-11T11:23:26.691 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.115354.log 2023-10-11T11:23:26.691 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.135870.log: /var/log/ceph/cluster2-client.mirror.115354.log: 80.5% -- replaced with /var/log/ceph/cluster2-client.mirror.115354.log.gz 2023-10-11T11:23:26.692 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124029.log 2023-10-11T11:23:26.693 INFO:teuthology.orchestra.run.smithi111.stderr: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.135870.log.gz 2023-10-11T11:23:26.694 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.124029.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.124029.log.gz 2023-10-11T11:23:26.694 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.135540.log 2023-10-11T11:23:26.695 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.135540.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.135540.log.gz 2023-10-11T11:23:26.695 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.140516.log 2023-10-11T11:23:26.696 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.140516.log: 87.1% -- replaced with /var/log/ceph/cluster1-client.mirror.140516.log.gz 2023-10-11T11:23:26.696 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.120199.log 2023-10-11T11:23:26.697 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.120199.log: 2.6% -- replaced with /var/log/ceph/cluster2-client.mirror.120199.log.gz 2023-10-11T11:23:26.697 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.135622.log 2023-10-11T11:23:26.700 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.135622.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.135622.log.gz 2023-10-11T11:23:26.700 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.106038.log 2023-10-11T11:23:26.701 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.admin.106038.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.106038.log.gz 2023-10-11T11:23:26.702 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.2.106866.log 2023-10-11T11:23:26.704 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.2.106866.log: 91.6% -- replaced with /var/log/ceph/cluster2-client.mirror.2.106866.log.gz 2023-10-11T11:23:26.704 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.106130.log 2023-10-11T11:23:26.705 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.admin.106130.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.106130.log.gz 2023-10-11T11:23:26.705 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117700.log 2023-10-11T11:23:26.706 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.117700.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.117700.log.gz 2023-10-11T11:23:26.706 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.145051.log 2023-10-11T11:23:26.707 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.admin.145051.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.145051.log.gz 2023-10-11T11:23:26.707 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.144353.log 2023-10-11T11:23:26.708 INFO:teuthology.orchestra.run.smithi111.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.141742.log.gz 2023-10-11T11:23:26.713 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.111099.log 2023-10-11T11:23:26.714 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.admin.144353.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.144353.log.gz 2023-10-11T11:23:26.714 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.111099.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.111099.log.gz 2023-10-11T11:23:26.714 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111077.log 2023-10-11T11:23:26.715 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137490.log 2023-10-11T11:23:26.715 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.111077.log: /var/log/ceph/cluster1-client.mirror.137490.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.137490.log.gz 2023-10-11T11:23:26.716 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.1.110449.log 2023-10-11T11:23:26.719 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.1.110449.log: 93.1% -- replaced with /var/log/ceph/cluster2-client.mirror.1.110449.log.gz 2023-10-11T11:23:26.719 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.126383.log 2023-10-11T11:23:26.720 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.126383.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.126383.log.gz 2023-10-11T11:23:26.720 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119152.log 2023-10-11T11:23:26.720 INFO:teuthology.orchestra.run.smithi111.stderr: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.111077.log.gz 2023-10-11T11:23:26.721 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109165.log 2023-10-11T11:23:26.721 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.119152.log: 80.4% -- replaced with /var/log/ceph/cluster2-client.mirror.119152.log.gz 2023-10-11T11:23:26.726 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.109165.log: 93.0% -- replaced with /var/log/ceph/cluster1-client.mirror.109165.log.gz 2023-10-11T11:23:26.727 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112534.log 2023-10-11T11:23:26.728 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.114942.log 2023-10-11T11:23:26.728 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.112534.log: /var/log/ceph/cluster2-client.mirror.114942.log: 89.0% -- replaced with /var/log/ceph/cluster1-client.mirror.112534.log.gz 2023-10-11T11:23:26.737 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.128117.log 2023-10-11T11:23:26.739 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.128117.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.128117.log.gz 2023-10-11T11:23:26.740 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.106782.log 2023-10-11T11:23:26.741 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.106782.log: 89.4% -- replaced with /var/log/ceph/cluster1-client.mirror.106782.log.gz 2023-10-11T11:23:26.741 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123125.log 2023-10-11T11:23:26.742 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.123125.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.123125.log.gz 2023-10-11T11:23:26.742 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.134617.log 2023-10-11T11:23:26.742 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.134617.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.134617.log.gz 2023-10-11T11:23:26.743 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117343.log 2023-10-11T11:23:26.744 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.117343.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.117343.log.gz 2023-10-11T11:23:26.744 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.111815.log 2023-10-11T11:23:26.745 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.111815.log: 91.4% -- replaced with /var/log/ceph/cluster2-client.mirror.111815.log.gz 2023-10-11T11:23:26.745 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111954.log 2023-10-11T11:23:26.746 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.111954.log: 89.0% -- replaced with /var/log/ceph/cluster1-client.mirror.111954.log.gz 2023-10-11T11:23:26.746 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109100.log 2023-10-11T11:23:26.747 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.109100.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.109100.log.gz 2023-10-11T11:23:26.747 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136838.log 2023-10-11T11:23:26.747 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.136838.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.136838.log.gz 2023-10-11T11:23:26.748 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.113478.log 2023-10-11T11:23:26.754 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.113478.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.113478.log.gz 2023-10-11T11:23:26.754 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.123957.log 2023-10-11T11:23:26.755 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.123957.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.123957.log.gz 2023-10-11T11:23:26.755 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116961.log 2023-10-11T11:23:26.756 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.127743.log 2023-10-11T11:23:26.756 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.116961.log: /var/log/ceph/cluster2-client.mirror.127743.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.116961.log.gz 2023-10-11T11:23:26.756 INFO:teuthology.orchestra.run.smithi111.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.127743.log.gz 2023-10-11T11:23:26.757 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.129264.log 2023-10-11T11:23:26.758 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.142922.log 2023-10-11T11:23:26.758 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.129264.log: /var/log/ceph/cluster2-client.mirror.142922.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.129264.log.gz 2023-10-11T11:23:26.760 INFO:teuthology.orchestra.run.smithi111.stderr: 91.2% -- replaced with /var/log/ceph/cluster2-client.mirror.142922.log.gz 2023-10-11T11:23:26.760 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136388.log 2023-10-11T11:23:26.761 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.136388.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.136388.log.gz 2023-10-11T11:23:26.761 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.141621.log 2023-10-11T11:23:26.762 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124402.log 2023-10-11T11:23:26.762 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.141621.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.141621.log.gz 2023-10-11T11:23:26.763 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.124402.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.124402.log.gz 2023-10-11T11:23:26.763 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.115938.log 2023-10-11T11:23:26.764 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123101.log 2023-10-11T11:23:26.764 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.115938.log: /var/log/ceph/cluster1-client.mirror.123101.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.115938.log.gz 2023-10-11T11:23:26.764 INFO:teuthology.orchestra.run.smithi111.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.123101.log.gz 2023-10-11T11:23:26.765 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112671.log 2023-10-11T11:23:26.770 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.112671.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.112671.log.gz 2023-10-11T11:23:26.770 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.142051.log 2023-10-11T11:23:26.771 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.142051.log: 80.6% -- replaced with /var/log/ceph/cluster2-client.mirror.142051.log.gz 2023-10-11T11:23:26.771 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126859.log 2023-10-11T11:23:26.772 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.126859.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.126859.log.gz 2023-10-11T11:23:26.772 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117169.log 2023-10-11T11:23:26.773 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126431.log 2023-10-11T11:23:26.773 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.117169.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.117169.log.gz 2023-10-11T11:23:26.774 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.126431.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.126431.log.gz 2023-10-11T11:23:26.774 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-mon.a.log 2023-10-11T11:23:26.784 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-mon.a.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.110742.log 2023-10-11T11:23:26.785 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.admin.110742.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.admin.110742.log.gz 2023-10-11T11:23:26.785 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.128158.log 2023-10-11T11:23:26.787 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.128158.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.128158.log.gz 2023-10-11T11:23:26.788 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.133884.log 2023-10-11T11:23:26.792 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.133884.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.133884.log.gz 2023-10-11T11:23:26.792 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.133193.log 2023-10-11T11:23:26.794 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.133193.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.133193.log.gz 2023-10-11T11:23:26.794 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.123493.log 2023-10-11T11:23:26.795 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.123493.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.123493.log.gz 2023-10-11T11:23:26.795 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.142219.log 2023-10-11T11:23:26.796 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.142219.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.142219.log.gz 2023-10-11T11:23:26.796 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.140135.log 2023-10-11T11:23:26.797 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.140135.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.140135.log.gz 2023-10-11T11:23:26.797 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.134489.log 2023-10-11T11:23:26.798 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.134489.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.134489.log.gz 2023-10-11T11:23:26.798 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.140219.log 2023-10-11T11:23:26.802 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.0.140219.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111593.log 2023-10-11T11:23:26.803 INFO:teuthology.orchestra.run.smithi111.stderr: 93.1% -- replaced with /var/log/ceph/cluster1-client.mirror.0.140219.log.gz 2023-10-11T11:23:26.809 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.111593.log: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.111593.log.gz 2023-10-11T11:23:26.809 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107245.log 2023-10-11T11:23:26.810 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.107245.log: 3.5% -- replaced with /var/log/ceph/cluster1-client.mirror.107245.log.gz 2023-10-11T11:23:26.810 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.130470.log 2023-10-11T11:23:26.811 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.130470.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.130470.log.gz 2023-10-11T11:23:26.811 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116509.log 2023-10-11T11:23:26.812 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.116509.log: 80.4% -- replaced with /var/log/ceph/cluster2-client.mirror.116509.log.gz 2023-10-11T11:23:26.812 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110025.log 2023-10-11T11:23:26.818 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.110025.log: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.110025.log.gz 2023-10-11T11:23:26.818 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.146433.log 2023-10-11T11:23:26.819 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.admin.146433.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.146433.log.gz 2023-10-11T11:23:26.819 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.125221.log 2023-10-11T11:23:26.820 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.125221.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.125221.log.gz 2023-10-11T11:23:26.820 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117149.log 2023-10-11T11:23:26.822 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.117149.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.128699.log 2023-10-11T11:23:26.822 INFO:teuthology.orchestra.run.smithi111.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.117149.log.gz 2023-10-11T11:23:26.823 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.128699.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.128699.log.gz 2023-10-11T11:23:26.823 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118796.log 2023-10-11T11:23:26.824 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.118796.log: 80.3% -- replaced with /var/log/ceph/cluster2-client.mirror.118796.log.gz 2023-10-11T11:23:26.824 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.109559.log 2023-10-11T11:23:26.826 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.109559.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.132802.log 2023-10-11T11:23:26.830 INFO:teuthology.orchestra.run.smithi111.stderr: 93.3% -- replaced with /var/log/ceph/cluster2-client.mirror.109559.log.gz 2023-10-11T11:23:26.831 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.132802.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.132802.log.gz 2023-10-11T11:23:26.832 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117722.log 2023-10-11T11:23:26.834 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.117722.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.141855.log 2023-10-11T11:23:26.850 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.141855.log: 77.2% -- replaced with /var/log/ceph/cluster2-client.mirror.141855.log.gz 2023-10-11T11:23:26.850 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.141763.log 2023-10-11T11:23:26.861 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.141763.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.141763.log.gz 2023-10-11T11:23:26.861 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.133360.log 2023-10-11T11:23:26.872 INFO:teuthology.orchestra.run.smithi111.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.118393.log.gz 2023-10-11T11:23:26.872 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.133360.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116398.log 2023-10-11T11:23:26.872 INFO:teuthology.orchestra.run.smithi111.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.133360.log.gz 2023-10-11T11:23:26.872 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.116398.log: 80.2% -- replaced with /var/log/ceph/cluster2-client.mirror.116398.log.gz 2023-10-11T11:23:26.873 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.133822.log 2023-10-11T11:23:26.894 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.133822.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.133822.log.gz 2023-10-11T11:23:26.894 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.125086.log 2023-10-11T11:23:26.894 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.0.125086.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.0.125086.log.gz 2023-10-11T11:23:26.905 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.134427.log 2023-10-11T11:23:26.905 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.134427.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.134427.log.gz 2023-10-11T11:23:26.905 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.114752.log 2023-10-11T11:23:26.905 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.114752.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.114752.log.gz 2023-10-11T11:23:26.906 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117003.log 2023-10-11T11:23:26.916 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.117003.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.123705.log 2023-10-11T11:23:26.922 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.123705.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.123705.log.gz 2023-10-11T11:23:26.922 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.115603.log 2023-10-11T11:23:26.932 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.115603.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.115603.log.gz 2023-10-11T11:23:26.933 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113015.log 2023-10-11T11:23:26.945 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.131538.log 2023-10-11T11:23:26.945 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.113015.log: 89.4% -- replaced with /var/log/ceph/cluster1-client.mirror.113015.log.gz 2023-10-11T11:23:26.958 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.131538.log: 93.1% -- replaced with /var/log/ceph/cluster2-client.mirror.131538.log.gz 2023-10-11T11:23:26.958 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.133554.log 2023-10-11T11:23:26.972 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.133554.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.133554.log.gz 2023-10-11T11:23:26.972 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.109004.log 2023-10-11T11:23:26.986 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.109004.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.109004.log.gz 2023-10-11T11:23:26.986 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.141341.log 2023-10-11T11:23:27.000 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.141341.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.141341.log.gz 2023-10-11T11:23:27.000 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.114813.log 2023-10-11T11:23:27.011 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.114813.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.110583.log 2023-10-11T11:23:27.022 INFO:teuthology.orchestra.run.smithi111.stderr: 90.9% -- replaced with /var/log/ceph/cluster2-client.mirror.114813.log.gz 2023-10-11T11:23:27.033 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.110583.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.110583.log.gz 2023-10-11T11:23:27.033 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137085.log 2023-10-11T11:23:27.054 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.137085.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.137085.log.gz 2023-10-11T11:23:27.054 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121459.log 2023-10-11T11:23:27.064 INFO:teuthology.orchestra.run.smithi111.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.114942.log.gz 2023-10-11T11:23:27.065 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.121459.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.127428.log 2023-10-11T11:23:27.065 INFO:teuthology.orchestra.run.smithi111.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.121459.log.gz 2023-10-11T11:23:27.068 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.127428.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.140474.log.gz 2023-10-11T11:23:27.068 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.140993.log 2023-10-11T11:23:27.068 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.140993.log: 92.3% -- replaced with /var/log/ceph/cluster2-client.mirror.127428.log.gz 2023-10-11T11:23:27.068 INFO:teuthology.orchestra.run.smithi111.stderr: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.140993.log.gz 2023-10-11T11:23:27.068 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136635.log 2023-10-11T11:23:27.069 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.136635.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.136635.log.gz 2023-10-11T11:23:27.070 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.142587.log 2023-10-11T11:23:27.070 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.142587.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.142587.log.gz 2023-10-11T11:23:27.070 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.134362.log 2023-10-11T11:23:27.071 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.134362.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.134362.log.gz 2023-10-11T11:23:27.071 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.131331.log 2023-10-11T11:23:27.072 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.131331.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.129401.log 2023-10-11T11:23:27.074 INFO:teuthology.orchestra.run.smithi111.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.131331.log.gz 2023-10-11T11:23:27.074 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.129401.log: 92.2% -- replaced with /var/log/ceph/cluster2-client.mirror.129401.log.gz 2023-10-11T11:23:27.074 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.135209.log 2023-10-11T11:23:27.075 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.135209.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.135209.log.gz 2023-10-11T11:23:27.075 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116025.log 2023-10-11T11:23:27.077 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.116025.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.116025.log.gz 2023-10-11T11:23:27.077 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119960.log 2023-10-11T11:23:27.078 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.119960.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.119960.log.gz 2023-10-11T11:23:27.078 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.108334.log 2023-10-11T11:23:27.079 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.admin.108334.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.108334.log.gz 2023-10-11T11:23:27.079 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.106176.log 2023-10-11T11:23:27.080 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.admin.106176.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.106176.log.gz 2023-10-11T11:23:27.080 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120971.log 2023-10-11T11:23:27.081 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.120971.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.120971.log.gz 2023-10-11T11:23:27.081 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116863.log 2023-10-11T11:23:27.081 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.116863.log: 80.5% -- replaced with /var/log/ceph/cluster2-client.mirror.116863.log.gz 2023-10-11T11:23:27.082 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.112066.log 2023-10-11T11:23:27.087 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.112066.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.112066.log.gz 2023-10-11T11:23:27.088 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.130720.log 2023-10-11T11:23:27.088 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.130720.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.130720.log.gz 2023-10-11T11:23:27.089 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137063.log 2023-10-11T11:23:27.089 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.137063.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.137063.log.gz 2023-10-11T11:23:27.089 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.123378.log 2023-10-11T11:23:27.091 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.123378.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.123378.log.gz 2023-10-11T11:23:27.092 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.109252.log 2023-10-11T11:23:27.096 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.109252.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.117003.log.gz 2023-10-11T11:23:27.096 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137220.log 2023-10-11T11:23:27.097 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.137220.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.137220.log.gz 2023-10-11T11:23:27.097 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121268.log 2023-10-11T11:23:27.099 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.121268.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.121268.log.gz 2023-10-11T11:23:27.099 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120804.log 2023-10-11T11:23:27.101 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.120804.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.120804.log.gz 2023-10-11T11:23:27.102 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138503.log 2023-10-11T11:23:27.102 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.114712.log 2023-10-11T11:23:27.103 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.138503.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.138503.log.gz 2023-10-11T11:23:27.103 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.114712.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.114712.log.gz 2023-10-11T11:23:27.104 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.135230.log 2023-10-11T11:23:27.105 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137108.log 2023-10-11T11:23:27.105 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.135230.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.135230.log.gz 2023-10-11T11:23:27.105 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.137108.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.137108.log.gz 2023-10-11T11:23:27.106 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126955.log 2023-10-11T11:23:27.106 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.126955.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.126955.log.gz 2023-10-11T11:23:27.106 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.100513.log 2023-10-11T11:23:27.107 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.admin.100513.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.100513.log.gz 2023-10-11T11:23:27.107 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.126316.log 2023-10-11T11:23:27.108 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.126316.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.126316.log.gz 2023-10-11T11:23:27.109 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.129308.log 2023-10-11T11:23:27.109 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.129308.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.129308.log.gz 2023-10-11T11:23:27.110 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124640.log 2023-10-11T11:23:27.111 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.124640.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.124640.log.gz 2023-10-11T11:23:27.111 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137558.log 2023-10-11T11:23:27.112 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.137558.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.137558.log.gz 2023-10-11T11:23:27.112 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.129558.log 2023-10-11T11:23:27.113 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.129558.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.129558.log.gz 2023-10-11T11:23:27.113 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.114672.log 2023-10-11T11:23:27.114 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.114672.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.114672.log.gz 2023-10-11T11:23:27.115 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.145741.log 2023-10-11T11:23:27.116 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.admin.145741.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.145741.log.gz 2023-10-11T11:23:27.116 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.114732.log 2023-10-11T11:23:27.116 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.114732.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.114732.log.gz 2023-10-11T11:23:27.117 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110449.log 2023-10-11T11:23:27.118 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.110449.log: 81.3% -- replaced with /var/log/ceph/cluster1-client.mirror.110449.log.gz 2023-10-11T11:23:27.118 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.107382.log 2023-10-11T11:23:27.119 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118112.log 2023-10-11T11:23:27.119 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.admin.107382.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.107382.log.gz 2023-10-11T11:23:27.119 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.144583.log 2023-10-11T11:23:27.120 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.118112.log: /var/log/ceph/cluster2-client.admin.144583.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.144583.log.gz 2023-10-11T11:23:27.120 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136344.log 2023-10-11T11:23:27.120 INFO:teuthology.orchestra.run.smithi111.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.118112.log.gz 2023-10-11T11:23:27.121 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.136344.log: 2.6% -- replaced with /var/log/ceph/cluster1-client.mirror.136344.log.gz 2023-10-11T11:23:27.121 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.109145.log 2023-10-11T11:23:27.126 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.109145.log: 93.1% -- replaced with /var/log/ceph/cluster2-client.mirror.109145.log.gz 2023-10-11T11:23:27.126 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137130.log 2023-10-11T11:23:27.127 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.137130.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.137130.log.gz 2023-10-11T11:23:27.127 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.133298.log 2023-10-11T11:23:27.128 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.133298.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.133298.log.gz 2023-10-11T11:23:27.128 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.127018.log 2023-10-11T11:23:27.130 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.127018.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.127018.log.gz 2023-10-11T11:23:27.130 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125086.log 2023-10-11T11:23:27.131 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.125086.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.125086.log.gz 2023-10-11T11:23:27.131 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119288.log 2023-10-11T11:23:27.132 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.119288.log: 80.2% -- replaced with /var/log/ceph/cluster2-client.mirror.119288.log.gz 2023-10-11T11:23:27.132 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.143985.log 2023-10-11T11:23:27.133 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.admin.143985.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.143985.log.gz 2023-10-11T11:23:27.133 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.130176.log 2023-10-11T11:23:27.135 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.130176.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.130176.log.gz 2023-10-11T11:23:27.135 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.144445.log 2023-10-11T11:23:27.136 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.admin.144445.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.144445.log.gz 2023-10-11T11:23:27.136 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.141403.log 2023-10-11T11:23:27.139 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.141403.log: 91.2% -- replaced with /var/log/ceph/cluster2-client.mirror.141403.log.gz 2023-10-11T11:23:27.139 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.130902.log 2023-10-11T11:23:27.141 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.130902.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.130902.log.gz 2023-10-11T11:23:27.141 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.134258.log 2023-10-11T11:23:27.144 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.134258.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.134258.log.gz 2023-10-11T11:23:27.145 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136950.log 2023-10-11T11:23:27.145 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.136950.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.136950.log.gz 2023-10-11T11:23:27.146 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124245.log 2023-10-11T11:23:27.146 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.124245.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.124245.log.gz 2023-10-11T11:23:27.147 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.135643.log 2023-10-11T11:23:27.148 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.135643.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.135643.log.gz 2023-10-11T11:23:27.148 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.128747.log 2023-10-11T11:23:27.149 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.128747.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.128747.log.gz 2023-10-11T11:23:27.149 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.130402.log 2023-10-11T11:23:27.151 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.130402.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.130402.log.gz 2023-10-11T11:23:27.151 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.142474.log 2023-10-11T11:23:27.154 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.0.142474.log: 91.5% -- replaced with /var/log/ceph/cluster1-client.mirror.0.142474.log.gz 2023-10-11T11:23:27.154 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.140783.log 2023-10-11T11:23:27.156 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.140783.log: 91.1% -- replaced with /var/log/ceph/cluster2-client.mirror.140783.log.gz 2023-10-11T11:23:27.157 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.121684.log 2023-10-11T11:23:27.158 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.121684.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.121684.log.gz 2023-10-11T11:23:27.158 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.136056.log 2023-10-11T11:23:27.160 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.136056.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.117722.log.gz 2023-10-11T11:23:27.161 INFO:teuthology.orchestra.run.smithi111.stderr: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.136056.log.gz 2023-10-11T11:23:27.161 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109514.log 2023-10-11T11:23:27.162 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.144959.log 2023-10-11T11:23:27.162 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.109514.log: /var/log/ceph/cluster2-client.admin.144959.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.109514.log.gz 2023-10-11T11:23:27.163 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.144959.log.gz 2023-10-11T11:23:27.163 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.128406.log 2023-10-11T11:23:27.164 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.145835.log 2023-10-11T11:23:27.164 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.128406.log: /var/log/ceph/cluster2-client.admin.145835.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.145835.log.gz 2023-10-11T11:23:27.164 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.142010.log 2023-10-11T11:23:27.165 INFO:teuthology.orchestra.run.smithi111.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.128406.log.gz 2023-10-11T11:23:27.165 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.121930.log 2023-10-11T11:23:27.165 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.142010.log: /var/log/ceph/cluster2-client.mirror.121930.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.142010.log.gz 2023-10-11T11:23:27.166 INFO:teuthology.orchestra.run.smithi111.stderr: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.121930.log.gz 2023-10-11T11:23:27.166 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.132039.log 2023-10-11T11:23:27.167 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136815.log 2023-10-11T11:23:27.167 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.132039.log: /var/log/ceph/cluster1-client.mirror.136815.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.132039.log.gz 2023-10-11T11:23:27.167 INFO:teuthology.orchestra.run.smithi111.stderr: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.136815.log.gz 2023-10-11T11:23:27.168 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124530.log 2023-10-11T11:23:27.169 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136928.log 2023-10-11T11:23:27.169 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.124530.log: /var/log/ceph/cluster1-client.mirror.136928.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.124530.log.gz 2023-10-11T11:23:27.169 INFO:teuthology.orchestra.run.smithi111.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.136928.log.gz 2023-10-11T11:23:27.169 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.144813.log 2023-10-11T11:23:27.170 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136478.log 2023-10-11T11:23:27.170 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.admin.144813.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.144813.log.gz 2023-10-11T11:23:27.171 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.136478.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.136478.log.gz 2023-10-11T11:23:27.171 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137648.log 2023-10-11T11:23:27.171 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126996.log 2023-10-11T11:23:27.172 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.137648.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.137648.log.gz 2023-10-11T11:23:27.172 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.126996.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.126996.log.gz 2023-10-11T11:23:27.172 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113865.log 2023-10-11T11:23:27.173 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.1.112374.log 2023-10-11T11:23:27.175 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.113865.log: /var/log/ceph/cluster2-client.mirror.1.112374.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.1.112374.log.gz 2023-10-11T11:23:27.175 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.140560.log 2023-10-11T11:23:27.177 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.140560.log: 91.4% -- replaced with /var/log/ceph/cluster2-client.mirror.140560.log.gz 2023-10-11T11:23:27.177 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124777.log 2023-10-11T11:23:27.178 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.124777.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.113865.log.gz 2023-10-11T11:23:27.179 INFO:teuthology.orchestra.run.smithi111.stderr: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.124777.log.gz 2023-10-11T11:23:27.180 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.135746.log 2023-10-11T11:23:27.181 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.106698.log 2023-10-11T11:23:27.181 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.135746.log: /var/log/ceph/cluster2-client.mirror.106698.log: 73.5% -- replaced with /var/log/ceph/cluster2-client.mirror.106698.log.gz 2023-10-11T11:23:27.181 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.138711.log 2023-10-11T11:23:27.182 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.138711.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.135746.log.gz 2023-10-11T11:23:27.193 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.139802.log 2023-10-11T11:23:27.194 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.141424.log 2023-10-11T11:23:27.194 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.139802.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.139802.log.gz 2023-10-11T11:23:27.195 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.141424.log: 80.6% -- replaced with /var/log/ceph/cluster2-client.mirror.141424.log.gz 2023-10-11T11:23:27.195 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.105854.log 2023-10-11T11:23:27.196 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.admin.105854.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.105854.log.gz 2023-10-11T11:23:27.196 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.135333.log 2023-10-11T11:23:27.196 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.135333.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.135333.log.gz 2023-10-11T11:23:27.197 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126479.log 2023-10-11T11:23:27.197 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.126479.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.126479.log.gz 2023-10-11T11:23:27.198 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116633.log 2023-10-11T11:23:27.198 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.116633.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.116633.log.gz 2023-10-11T11:23:27.198 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124085.log 2023-10-11T11:23:27.199 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.124085.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.124085.log.gz 2023-10-11T11:23:27.199 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.106452.log 2023-10-11T11:23:27.200 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.admin.106452.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.106452.log.gz 2023-10-11T11:23:27.200 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.133595.log 2023-10-11T11:23:27.201 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.133595.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.133595.log.gz 2023-10-11T11:23:27.202 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.111680.log 2023-10-11T11:23:27.207 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.111680.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.133028.log 2023-10-11T11:23:27.208 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.133028.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.133028.log.gz 2023-10-11T11:23:27.208 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.133504.log 2023-10-11T11:23:27.211 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.133504.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.113192.log 2023-10-11T11:23:27.211 INFO:teuthology.orchestra.run.smithi111.stderr: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.133504.log.gz 2023-10-11T11:23:27.213 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.113192.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.113192.log.gz 2023-10-11T11:23:27.213 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.141383.log 2023-10-11T11:23:27.214 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.141383.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.141383.log.gz 2023-10-11T11:23:27.214 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.128521.log 2023-10-11T11:23:27.215 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.128521.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.128521.log.gz 2023-10-11T11:23:27.216 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137895.log 2023-10-11T11:23:27.216 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.137895.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.137895.log.gz 2023-10-11T11:23:27.216 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.121480.log 2023-10-11T11:23:27.219 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.121480.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.121480.log.gz 2023-10-11T11:23:27.219 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.106084.log 2023-10-11T11:23:27.220 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.admin.106084.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.106084.log.gz 2023-10-11T11:23:27.220 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.132965.log 2023-10-11T11:23:27.221 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.132965.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.132965.log.gz 2023-10-11T11:23:27.221 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.142131.log 2023-10-11T11:23:27.222 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.142131.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.142131.log.gz 2023-10-11T11:23:27.223 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118956.log 2023-10-11T11:23:27.223 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.118956.log: 80.2% -- replaced with /var/log/ceph/cluster2-client.mirror.118956.log.gz 2023-10-11T11:23:27.223 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125769.log 2023-10-11T11:23:27.224 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.125769.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.125769.log.gz 2023-10-11T11:23:27.225 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.134555.log 2023-10-11T11:23:27.226 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.134555.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.134555.log.gz 2023-10-11T11:23:27.226 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124571.log 2023-10-11T11:23:27.228 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.124571.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.124571.log.gz 2023-10-11T11:23:27.228 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.131422.log 2023-10-11T11:23:27.229 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.131422.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.131422.log.gz 2023-10-11T11:23:27.230 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.115752.log 2023-10-11T11:23:27.238 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.1.115752.log: 92.9% -- replaced with /var/log/ceph/cluster1-client.mirror.1.115752.log.gz 2023-10-11T11:23:27.239 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.142564.log 2023-10-11T11:23:27.239 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.142564.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.142564.log.gz 2023-10-11T11:23:27.240 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.115879.log 2023-10-11T11:23:27.241 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.115879.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.115879.log.gz 2023-10-11T11:23:27.241 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.145881.log 2023-10-11T11:23:27.241 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.admin.145881.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.145881.log.gz 2023-10-11T11:23:27.242 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.130066.log 2023-10-11T11:23:27.243 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.130066.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.130066.log.gz 2023-10-11T11:23:27.243 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118734.log 2023-10-11T11:23:27.245 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.118734.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.118734.log.gz 2023-10-11T11:23:27.245 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124684.log 2023-10-11T11:23:27.246 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.124684.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.124684.log.gz 2023-10-11T11:23:27.246 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136658.log 2023-10-11T11:23:27.247 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.136658.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.136658.log.gz 2023-10-11T11:23:27.247 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.125856.log 2023-10-11T11:23:27.248 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.125856.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.125856.log.gz 2023-10-11T11:23:27.249 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.144913.log 2023-10-11T11:23:27.249 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.admin.144913.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.144913.log.gz 2023-10-11T11:23:27.249 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122565.log 2023-10-11T11:23:27.250 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.122565.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.122565.log.gz 2023-10-11T11:23:27.250 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136545.log 2023-10-11T11:23:27.251 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.136545.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.136545.log.gz 2023-10-11T11:23:27.251 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.136248.log 2023-10-11T11:23:27.253 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.136248.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.136248.log.gz 2023-10-11T11:23:27.253 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125721.log 2023-10-11T11:23:27.254 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.125721.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.125721.log.gz 2023-10-11T11:23:27.254 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122821.log 2023-10-11T11:23:27.255 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.122821.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.122821.log.gz 2023-10-11T11:23:27.255 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122446.log 2023-10-11T11:23:27.258 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.122446.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.122446.log.gz 2023-10-11T11:23:27.258 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.121111.log 2023-10-11T11:23:27.259 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.121111.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.121111.log.gz 2023-10-11T11:23:27.260 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.106755.log 2023-10-11T11:23:27.260 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.106755.log: 81.5% -- replaced with /var/log/ceph/cluster2-client.mirror.106755.log.gz 2023-10-11T11:23:27.261 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127674.log 2023-10-11T11:23:27.263 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.127674.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.127674.log.gz 2023-10-11T11:23:27.263 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121411.log 2023-10-11T11:23:27.264 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.121411.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.121411.log.gz 2023-10-11T11:23:27.264 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.107495.log 2023-10-11T11:23:27.265 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.107495.log: 4.7% -- replaced with /var/log/ceph/cluster2-client.mirror.107495.log.gz 2023-10-11T11:23:27.265 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.129106.log 2023-10-11T11:23:27.266 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.129106.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.129106.log.gz 2023-10-11T11:23:27.266 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.129490.log 2023-10-11T11:23:27.268 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.129490.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.129490.log.gz 2023-10-11T11:23:27.268 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.142729.log 2023-10-11T11:23:27.273 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.142729.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124482.log 2023-10-11T11:23:27.303 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.132826.log 2023-10-11T11:23:27.310 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.132826.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.132826.log.gz 2023-10-11T11:23:27.310 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113455.log 2023-10-11T11:23:27.310 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.124482.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.124482.log.gz 2023-10-11T11:23:27.331 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.113455.log: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.113455.log.gz 2023-10-11T11:23:27.331 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.143000.log 2023-10-11T11:23:27.342 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.143000.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.143000.log.gz 2023-10-11T11:23:27.342 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136793.log 2023-10-11T11:23:27.358 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.136793.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.136793.log.gz 2023-10-11T11:23:27.369 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.129130.log 2023-10-11T11:23:27.380 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.129130.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.129130.log.gz 2023-10-11T11:23:27.380 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137670.log 2023-10-11T11:23:27.390 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.137670.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.137670.log.gz 2023-10-11T11:23:27.391 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.142787.log 2023-10-11T11:23:27.402 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.142787.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.142787.log.gz 2023-10-11T11:23:27.402 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120763.log 2023-10-11T11:23:27.413 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.120763.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.120763.log.gz 2023-10-11T11:23:27.413 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.114592.log 2023-10-11T11:23:27.420 INFO:teuthology.orchestra.run.smithi111.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.111680.log.gz 2023-10-11T11:23:27.422 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.114592.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.114592.log.gz 2023-10-11T11:23:27.422 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.130042.log 2023-10-11T11:23:27.423 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.130042.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.130042.log.gz 2023-10-11T11:23:27.423 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111552.log 2023-10-11T11:23:27.424 INFO:teuthology.orchestra.run.smithi111.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.138711.log.gz 2023-10-11T11:23:27.424 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.111552.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.111552.log.gz 2023-10-11T11:23:27.424 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118934.log 2023-10-11T11:23:27.434 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.118934.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118638.log 2023-10-11T11:23:27.436 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.118638.log: 80.2% -- replaced with /var/log/ceph/cluster2-client.mirror.118638.log.gz 2023-10-11T11:23:27.446 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.127696.log 2023-10-11T11:23:27.447 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.127696.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.127696.log.gz 2023-10-11T11:23:27.456 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.129534.log 2023-10-11T11:23:27.458 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.129534.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.129534.log.gz 2023-10-11T11:23:27.467 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.145327.log 2023-10-11T11:23:27.473 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.admin.145327.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.145327.log.gz 2023-10-11T11:23:27.478 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.130087.log 2023-10-11T11:23:27.485 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.130087.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.130087.log.gz 2023-10-11T11:23:27.485 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124756.log 2023-10-11T11:23:27.487 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.124756.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.124756.log.gz 2023-10-11T11:23:27.487 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108868.log 2023-10-11T11:23:27.487 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.108868.log: 89.6% -- replaced with /var/log/ceph/cluster1-client.mirror.108868.log.gz 2023-10-11T11:23:27.488 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.122025.log 2023-10-11T11:23:27.489 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.122025.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.122025.log.gz 2023-10-11T11:23:27.489 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119627.log 2023-10-11T11:23:27.490 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.119627.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.119627.log.gz 2023-10-11T11:23:27.490 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112967.log 2023-10-11T11:23:27.492 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.112967.log: 89.3% -- replaced with /var/log/ceph/cluster1-client.mirror.112967.log.gz 2023-10-11T11:23:27.492 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113369.log 2023-10-11T11:23:27.498 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.113369.log: 93.0% -- replaced with /var/log/ceph/cluster1-client.mirror.113369.log.gz 2023-10-11T11:23:27.498 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.133906.log 2023-10-11T11:23:27.499 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.133906.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.133906.log.gz 2023-10-11T11:23:27.500 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.132677.log 2023-10-11T11:23:27.501 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.132677.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.132677.log.gz 2023-10-11T11:23:27.501 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.107602.log 2023-10-11T11:23:27.503 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.107602.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.107602.log.gz 2023-10-11T11:23:27.503 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.139967.log 2023-10-11T11:23:27.505 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.139967.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114835.log 2023-10-11T11:23:27.522 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.114835.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.114835.log.gz 2023-10-11T11:23:27.522 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.130494.log 2023-10-11T11:23:27.535 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.130494.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.130494.log.gz 2023-10-11T11:23:27.535 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.121565.log 2023-10-11T11:23:27.546 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.121565.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.131470.log 2023-10-11T11:23:27.557 INFO:teuthology.orchestra.run.smithi111.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.121565.log.gz 2023-10-11T11:23:27.568 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.131470.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.131470.log.gz 2023-10-11T11:23:27.579 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138143.log 2023-10-11T11:23:27.590 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.138143.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.138143.log.gz 2023-10-11T11:23:27.590 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118413.log 2023-10-11T11:23:27.601 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.118413.log: 80.3% -- replaced with /var/log/ceph/cluster2-client.mirror.118413.log.gz 2023-10-11T11:23:27.601 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.111161.log 2023-10-11T11:23:27.614 INFO:teuthology.orchestra.run.smithi111.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.109252.log.gz 2023-10-11T11:23:27.620 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.111161.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.111161.log.gz 2023-10-11T11:23:27.620 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.113349.log 2023-10-11T11:23:27.631 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.113349.log: 93.1% -- replaced with /var/log/ceph/cluster2-client.mirror.113349.log.gz 2023-10-11T11:23:27.631 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.145097.log 2023-10-11T11:23:27.631 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.admin.145097.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.145097.log.gz 2023-10-11T11:23:27.631 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.129873.log 2023-10-11T11:23:27.632 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.129873.log: 92.0% -- replaced with /var/log/ceph/cluster1-client.mirror.129873.log.gz 2023-10-11T11:23:27.632 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138323.log 2023-10-11T11:23:27.632 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.138323.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.138323.log.gz 2023-10-11T11:23:27.632 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114858.log 2023-10-11T11:23:27.632 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.114858.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.114858.log.gz 2023-10-11T11:23:27.633 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.114240.log 2023-10-11T11:23:27.633 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.114240.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.118934.log.gz 2023-10-11T11:23:27.634 INFO:teuthology.orchestra.run.smithi111.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.114240.log.gz 2023-10-11T11:23:27.634 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.146249.log 2023-10-11T11:23:27.635 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.admin.146249.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.146249.log.gz 2023-10-11T11:23:27.635 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137873.log 2023-10-11T11:23:27.635 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.141642.log 2023-10-11T11:23:27.635 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.137873.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.137873.log.gz 2023-10-11T11:23:27.635 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.141642.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.141642.log.gz 2023-10-11T11:23:27.636 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.125588.log 2023-10-11T11:23:27.637 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.125588.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.125588.log.gz 2023-10-11T11:23:27.637 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120739.log 2023-10-11T11:23:27.640 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138435.log 2023-10-11T11:23:27.646 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125495.log 2023-10-11T11:23:27.647 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.138435.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.138435.log.gz 2023-10-11T11:23:27.647 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.125495.log: /var/log/ceph/cluster1-client.mirror.120739.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.125495.log.gz 2023-10-11T11:23:27.648 INFO:teuthology.orchestra.run.smithi111.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.120739.log.gz 2023-10-11T11:23:27.649 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.144721.log 2023-10-11T11:23:27.650 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.admin.144721.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.144721.log.gz 2023-10-11T11:23:27.650 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118472.log 2023-10-11T11:23:27.650 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.118472.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.106782.log 2023-10-11T11:23:27.667 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.0.106782.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126407.log 2023-10-11T11:23:27.667 INFO:teuthology.orchestra.run.smithi111.stderr: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.0.106782.log.gz 2023-10-11T11:23:27.670 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.126407.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.126407.log.gz 2023-10-11T11:23:27.670 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126181.log 2023-10-11T11:23:27.671 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.126181.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.126181.log.gz 2023-10-11T11:23:27.680 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.130855.log 2023-10-11T11:23:27.682 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.130855.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.130855.log.gz 2023-10-11T11:23:27.692 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.108283.log 2023-10-11T11:23:27.703 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.108283.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.115187.log 2023-10-11T11:23:27.725 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.115187.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125543.log 2023-10-11T11:23:27.725 INFO:teuthology.orchestra.run.smithi111.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.115187.log.gz 2023-10-11T11:23:27.736 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.125543.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.125543.log.gz 2023-10-11T11:23:27.736 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119773.log 2023-10-11T11:23:27.750 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.119773.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137985.log 2023-10-11T11:23:27.750 INFO:teuthology.orchestra.run.smithi111.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.119773.log.gz 2023-10-11T11:23:27.761 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.137985.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.137985.log.gz 2023-10-11T11:23:27.761 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.113060.log 2023-10-11T11:23:27.772 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.113060.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.140841.log 2023-10-11T11:23:27.779 INFO:teuthology.orchestra.run.smithi111.stderr: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.113060.log.gz 2023-10-11T11:23:27.780 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.140841.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.142729.log.gz 2023-10-11T11:23:27.780 INFO:teuthology.orchestra.run.smithi111.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.140841.log.gz 2023-10-11T11:23:27.780 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.131244.log 2023-10-11T11:23:27.781 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.131244.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.131244.log.gz 2023-10-11T11:23:27.781 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137400.log 2023-10-11T11:23:27.783 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.137400.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.137400.log.gz 2023-10-11T11:23:27.783 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.136297.log 2023-10-11T11:23:27.787 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.136297.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.136297.log.gz 2023-10-11T11:23:27.787 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.129853.log 2023-10-11T11:23:27.789 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.129853.log: 92.2% -- replaced with /var/log/ceph/cluster2-client.mirror.129853.log.gz 2023-10-11T11:23:27.789 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.130313.log 2023-10-11T11:23:27.790 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.130313.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.147307.log 2023-10-11T11:23:27.791 INFO:teuthology.orchestra.run.smithi111.stderr: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.130313.log.gz 2023-10-11T11:23:27.792 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.admin.147307.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.147307.log.gz 2023-10-11T11:23:27.792 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.126542.log 2023-10-11T11:23:27.794 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.126542.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.126542.log.gz 2023-10-11T11:23:27.794 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112818.log 2023-10-11T11:23:27.795 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.112818.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.112818.log.gz 2023-10-11T11:23:27.795 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.132241.log 2023-10-11T11:23:27.796 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.132241.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.132241.log.gz 2023-10-11T11:23:27.796 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.128569.log 2023-10-11T11:23:27.797 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.128569.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.128569.log.gz 2023-10-11T11:23:27.798 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.2.121820.log 2023-10-11T11:23:27.802 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.2.121820.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.132443.log 2023-10-11T11:23:27.806 INFO:teuthology.orchestra.run.smithi111.stderr: 92.9% -- replaced with /var/log/ceph/cluster2-client.mirror.2.121820.log.gz 2023-10-11T11:23:27.807 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.132443.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.132443.log.gz 2023-10-11T11:23:27.807 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.140432.log 2023-10-11T11:23:27.809 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.140432.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.140432.log.gz 2023-10-11T11:23:27.809 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136770.log 2023-10-11T11:23:27.810 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.136770.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.136770.log.gz 2023-10-11T11:23:27.810 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123541.log 2023-10-11T11:23:27.811 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.123541.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.123541.log.gz 2023-10-11T11:23:27.811 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109076.log 2023-10-11T11:23:27.812 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.109076.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.109076.log.gz 2023-10-11T11:23:27.812 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137963.log 2023-10-11T11:23:27.813 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.137963.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.137963.log.gz 2023-10-11T11:23:27.813 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.143479.log 2023-10-11T11:23:27.814 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.admin.143479.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.143479.log.gz 2023-10-11T11:23:27.814 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.134957.log 2023-10-11T11:23:27.815 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.134957.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.134957.log.gz 2023-10-11T11:23:27.816 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.128364.log 2023-10-11T11:23:27.818 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.128364.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137783.log 2023-10-11T11:23:27.818 INFO:teuthology.orchestra.run.smithi111.stderr: 92.3% -- replaced with /var/log/ceph/cluster2-client.mirror.128364.log.gz 2023-10-11T11:23:27.819 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.137783.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.137783.log.gz 2023-10-11T11:23:27.819 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.141037.log 2023-10-11T11:23:27.821 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.141037.log: 91.4% -- replaced with /var/log/ceph/cluster2-client.mirror.141037.log.gz 2023-10-11T11:23:27.822 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110232.log 2023-10-11T11:23:27.823 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.110232.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.110232.log.gz 2023-10-11T11:23:27.823 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127109.log 2023-10-11T11:23:27.824 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.127109.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.127109.log.gz 2023-10-11T11:23:27.824 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126068.log 2023-10-11T11:23:27.825 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.126068.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.106222.log 2023-10-11T11:23:27.826 INFO:teuthology.orchestra.run.smithi111.stderr: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.126068.log.gz 2023-10-11T11:23:27.827 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.admin.106222.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.106222.log.gz 2023-10-11T11:23:27.827 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.133740.log 2023-10-11T11:23:27.828 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.133740.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.133740.log.gz 2023-10-11T11:23:27.829 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122173.log 2023-10-11T11:23:27.829 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.122173.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.122173.log.gz 2023-10-11T11:23:27.830 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.142110.log 2023-10-11T11:23:27.833 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.142110.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137423.log 2023-10-11T11:23:27.855 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.137423.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.137423.log.gz 2023-10-11T11:23:27.855 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137738.log 2023-10-11T11:23:27.871 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.137738.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.137738.log.gz 2023-10-11T11:23:27.871 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.135416.log 2023-10-11T11:23:27.882 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.135416.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.135416.log.gz 2023-10-11T11:23:27.882 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138210.log 2023-10-11T11:23:27.892 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.138210.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.138210.log.gz 2023-10-11T11:23:27.892 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.120623.log 2023-10-11T11:23:27.913 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.120623.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.120623.log.gz 2023-10-11T11:23:27.913 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118571.log 2023-10-11T11:23:27.924 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.118571.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.133678.log 87.5% 2023-10-11T11:23:27.924 INFO:teuthology.orchestra.run.smithi111.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.118571.log.gz 2023-10-11T11:23:27.938 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.133678.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.110798.log 2023-10-11T11:23:27.938 INFO:teuthology.orchestra.run.smithi111.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.133678.log.gz 2023-10-11T11:23:27.938 INFO:teuthology.orchestra.run.smithi111.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.108283.log.gz 2023-10-11T11:23:27.939 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.110798.log: 91.3% -- replaced with /var/log/ceph/cluster2-client.mirror.110798.log.gz 2023-10-11T11:23:27.939 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.112130.log 2023-10-11T11:23:27.950 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.112130.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137153.log 2023-10-11T11:23:27.962 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.137153.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.137153.log.gz 2023-10-11T11:23:27.962 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.129217.log 2023-10-11T11:23:27.973 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.129217.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.133843.log 2023-10-11T11:23:27.974 INFO:teuthology.orchestra.run.smithi111.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.129217.log.gz 2023-10-11T11:23:27.975 INFO:teuthology.orchestra.run.smithi111.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.139967.log.gz 2023-10-11T11:23:27.976 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.133843.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.133843.log.gz 2023-10-11T11:23:27.977 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.111404.log 2023-10-11T11:23:27.978 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.111404.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.111404.log.gz 2023-10-11T11:23:27.979 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.143939.log 2023-10-11T11:23:27.979 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.admin.143939.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.143939.log.gz 2023-10-11T11:23:27.979 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.113650.log 2023-10-11T11:23:27.986 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.113650.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.113650.log.gz 2023-10-11T11:23:27.986 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118132.log 2023-10-11T11:23:27.998 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.118132.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.118132.log.gz 2023-10-11T11:23:27.998 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120507.log 2023-10-11T11:23:28.002 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.120507.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.120507.log.gz 2023-10-11T11:23:28.002 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121179.log 2023-10-11T11:23:28.003 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.121179.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.121179.log.gz 2023-10-11T11:23:28.003 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124005.log 2023-10-11T11:23:28.003 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.124005.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.124005.log.gz 2023-10-11T11:23:28.003 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.123188.log 2023-10-11T11:23:28.004 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.123188.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117501.log 2023-10-11T11:23:28.004 INFO:teuthology.orchestra.run.smithi111.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.123188.log.gz 2023-10-11T11:23:28.004 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.117501.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.117501.log.gz 2023-10-11T11:23:28.005 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.142975.log 2023-10-11T11:23:28.005 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.142975.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.142975.log.gz 2023-10-11T11:23:28.005 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112086.log 2023-10-11T11:23:28.011 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.112086.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.110114.log 2023-10-11T11:23:28.012 INFO:teuthology.orchestra.run.smithi111.stderr: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.112086.log.gz 2023-10-11T11:23:28.018 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.110114.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.110114.log.gz 2023-10-11T11:23:28.018 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.141441.log 2023-10-11T11:23:28.019 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.141441.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.141441.log.gz 2023-10-11T11:23:28.020 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.100782.log 2023-10-11T11:23:28.020 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.admin.100782.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.100782.log.gz 2023-10-11T11:23:28.021 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.115662.log 2023-10-11T11:23:28.022 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.115662.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.143025.log 2023-10-11T11:23:28.022 INFO:teuthology.orchestra.run.smithi111.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.115662.log.gz 2023-10-11T11:23:28.023 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.143025.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.143025.log.gz 2023-10-11T11:23:28.024 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.127470.log 2023-10-11T11:23:28.025 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.127470.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.127470.log.gz 2023-10-11T11:23:28.025 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.132015.log 2023-10-11T11:23:28.026 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.132015.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.132015.log.gz 2023-10-11T11:23:28.027 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.130922.log 2023-10-11T11:23:28.027 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.130922.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.130922.log.gz 2023-10-11T11:23:28.028 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117663.log 2023-10-11T11:23:28.028 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.117663.log: 80.4% -- replaced with /var/log/ceph/cluster2-client.mirror.117663.log.gz 2023-10-11T11:23:28.029 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107330.log 2023-10-11T11:23:28.029 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.107330.log: 53.8% -- replaced with /var/log/ceph/cluster1-client.mirror.107330.log.gz 2023-10-11T11:23:28.030 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.2.112264.log 2023-10-11T11:23:28.034 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.2.112264.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.134852.log 2023-10-11T11:23:28.035 INFO:teuthology.orchestra.run.smithi111.stderr: 93.0% -- replaced with /var/log/ceph/cluster1-client.mirror.2.112264.log.gz 2023-10-11T11:23:28.037 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.134852.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.134852.log.gz 2023-10-11T11:23:28.037 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124148.log 2023-10-11T11:23:28.038 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.124148.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.124148.log.gz 2023-10-11T11:23:28.039 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.132108.log 2023-10-11T11:23:28.041 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.132108.log: 92.3% -- replaced with /var/log/ceph/cluster2-client.mirror.132108.log.gz 2023-10-11T11:23:28.041 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124221.log 2023-10-11T11:23:28.042 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.124221.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.124221.log.gz 2023-10-11T11:23:28.042 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.142196.log 2023-10-11T11:23:28.043 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.142196.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.142196.log.gz 2023-10-11T11:23:28.044 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.135891.log 2023-10-11T11:23:28.045 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.135891.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.135891.log.gz 2023-10-11T11:23:28.045 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.142240.log 2023-10-11T11:23:28.046 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.142240.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.142240.log.gz 2023-10-11T11:23:28.046 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123862.log 2023-10-11T11:23:28.048 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.123862.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.118472.log.gz 2023-10-11T11:23:28.049 INFO:teuthology.orchestra.run.smithi111.stderr: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.123862.log.gz 2023-10-11T11:23:28.049 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.122700.log 2023-10-11T11:23:28.050 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.122700.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.122700.log.gz 2023-10-11T11:23:28.050 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.129175.log 2023-10-11T11:23:28.051 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.129175.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.131971.log 2023-10-11T11:23:28.052 INFO:teuthology.orchestra.run.smithi111.stderr: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.129175.log.gz 2023-10-11T11:23:28.058 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.131971.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.131971.log.gz 2023-10-11T11:23:28.059 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117278.log 2023-10-11T11:23:28.068 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111906.log 2023-10-11T11:23:28.069 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.117278.log: /var/log/ceph/cluster1-client.mirror.111906.log: 88.8% -- replaced with /var/log/ceph/cluster1-client.mirror.111906.log.gz 2023-10-11T11:23:28.069 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137760.log 2023-10-11T11:23:28.069 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.137760.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.137760.log.gz 2023-10-11T11:23:28.070 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117385.log 2023-10-11T11:23:28.070 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.117385.log: 80.3% -- replaced with /var/log/ceph/cluster2-client.mirror.117385.log.gz 2023-10-11T11:23:28.070 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114174.log 2023-10-11T11:23:28.071 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.114174.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.114174.log.gz 2023-10-11T11:23:28.071 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127811.log 2023-10-11T11:23:28.074 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.127811.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.127811.log.gz 2023-10-11T11:23:28.074 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.123261.log 2023-10-11T11:23:28.075 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.123261.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.123261.log.gz 2023-10-11T11:23:28.075 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137445.log 2023-10-11T11:23:28.075 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.137445.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.142110.log.gz 2023-10-11T11:23:28.075 INFO:teuthology.orchestra.run.smithi111.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.137445.log.gz 2023-10-11T11:23:28.076 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123517.log 2023-10-11T11:23:28.076 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.123517.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.123517.log.gz 2023-10-11T11:23:28.076 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.112899.log 2023-10-11T11:23:28.077 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.112899.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.112899.log.gz 2023-10-11T11:23:28.077 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.132354.log 2023-10-11T11:23:28.079 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.132354.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.132354.log.gz 2023-10-11T11:23:28.079 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.133719.log 2023-10-11T11:23:28.079 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.133719.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.133719.log.gz 2023-10-11T11:23:28.080 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.135932.log 2023-10-11T11:23:28.080 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.135932.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.135932.log.gz 2023-10-11T11:23:28.081 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.142339.log 2023-10-11T11:23:28.082 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.142339.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.142339.log.gz 2023-10-11T11:23:28.083 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112198.log 2023-10-11T11:23:28.083 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.112198.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.112198.log.gz 2023-10-11T11:23:28.083 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.131837.log 2023-10-11T11:23:28.084 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.143709.log 2023-10-11T11:23:28.084 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.131837.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.131837.log.gz 2023-10-11T11:23:28.084 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.admin.143709.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.143709.log.gz 2023-10-11T11:23:28.085 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.121248.log 2023-10-11T11:23:28.086 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.133636.log 2023-10-11T11:23:28.086 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.121248.log: /var/log/ceph/cluster2-client.mirror.133636.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.121248.log.gz 2023-10-11T11:23:28.087 INFO:teuthology.orchestra.run.smithi111.stderr: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.133636.log.gz 2023-10-11T11:23:28.087 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.105670.log 2023-10-11T11:23:28.088 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.admin.105670.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.105670.log.gz 2023-10-11T11:23:28.088 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.129332.log 2023-10-11T11:23:28.089 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.129332.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.129332.log.gz 2023-10-11T11:23:28.089 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137580.log 2023-10-11T11:23:28.090 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.137580.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.137580.log.gz 2023-10-11T11:23:28.090 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.132289.log 2023-10-11T11:23:28.091 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.132289.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.132289.log.gz 2023-10-11T11:23:28.091 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122910.log 2023-10-11T11:23:28.093 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.122910.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.122910.log.gz 2023-10-11T11:23:28.093 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120262.log 2023-10-11T11:23:28.094 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.120262.log: 1.3% -- replaced with /var/log/ceph/cluster1-client.mirror.120262.log.gz 2023-10-11T11:23:28.094 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116675.log 2023-10-11T11:23:28.095 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.116675.log: 80.5% -- replaced with /var/log/ceph/cluster2-client.mirror.116675.log.gz 2023-10-11T11:23:28.095 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124660.log 2023-10-11T11:23:28.096 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.124660.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.124660.log.gz 2023-10-11T11:23:28.096 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.107444.log 2023-10-11T11:23:28.096 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.admin.107444.log: 66.4% -- replaced with /var/log/ceph/cluster2-client.admin.107444.log.gz 2023-10-11T11:23:28.097 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123309.log 2023-10-11T11:23:28.097 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.123309.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.123309.log.gz 2023-10-11T11:23:28.098 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.140158.log 2023-10-11T11:23:28.098 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.140158.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.140158.log.gz 2023-10-11T11:23:28.098 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.132939.log 2023-10-11T11:23:28.101 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.132939.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.132939.log.gz 2023-10-11T11:23:28.101 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.141210.log 2023-10-11T11:23:28.101 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.141210.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.141210.log.gz 2023-10-11T11:23:28.102 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.142670.log 2023-10-11T11:23:28.102 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.142670.log: 80.5% -- replaced with /var/log/ceph/cluster2-client.mirror.142670.log.gz 2023-10-11T11:23:28.103 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.145235.log 2023-10-11T11:23:28.103 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.admin.145235.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.145235.log.gz 2023-10-11T11:23:28.104 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.144675.log 2023-10-11T11:23:28.104 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.130333.log 2023-10-11T11:23:28.104 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.admin.144675.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.144675.log.gz 2023-10-11T11:23:28.106 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.130333.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.130333.log.gz 2023-10-11T11:23:28.106 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.106406.log 2023-10-11T11:23:28.106 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.admin.106406.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.106406.log.gz 2023-10-11T11:23:28.107 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118233.log 2023-10-11T11:23:28.112 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.118233.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.140539.log 2023-10-11T11:23:28.114 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.140539.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.140539.log.gz 2023-10-11T11:23:28.114 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138188.log 2023-10-11T11:23:28.115 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.138188.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.138188.log.gz 2023-10-11T11:23:28.115 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.134679.log 2023-10-11T11:23:28.116 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.134679.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.134679.log.gz 2023-10-11T11:23:28.116 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.110178.log 2023-10-11T11:23:28.127 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.110178.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.123725.log 2023-10-11T11:23:28.148 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.123725.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.123725.log.gz 2023-10-11T11:23:28.148 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.138689.log 2023-10-11T11:23:28.159 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.138689.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.138689.log.gz 2023-10-11T11:23:28.159 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.132986.log 2023-10-11T11:23:28.170 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.132986.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.132986.log.gz 2023-10-11T11:23:28.170 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117191.log 2023-10-11T11:23:28.183 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.117191.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.131172.log 2023-10-11T11:23:28.196 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.131172.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.131172.log.gz 2023-10-11T11:23:28.196 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113236.log 2023-10-11T11:23:28.218 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.113236.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116282.log 2023-10-11T11:23:28.218 INFO:teuthology.orchestra.run.smithi111.stderr: 89.4% -- replaced with /var/log/ceph/cluster1-client.mirror.113236.log.gz 2023-10-11T11:23:28.219 INFO:teuthology.orchestra.run.smithi111.stderr: 90.1% -- replaced with /var/log/ceph/cluster2-mon.a.log.gz 2023-10-11T11:23:28.229 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.116282.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.120826.log 2023-10-11T11:23:28.229 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.120826.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.120826.log.gz 2023-10-11T11:23:28.230 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.146525.log 2023-10-11T11:23:28.240 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.admin.146525.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.146525.log.gz 2023-10-11T11:23:28.240 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.120367.log 2023-10-11T11:23:28.251 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.120367.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.120367.log.gz 2023-10-11T11:23:28.251 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121227.log 2023-10-11T11:23:28.273 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.121227.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.121227.log.gz 2023-10-11T11:23:28.273 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124126.log 2023-10-11T11:23:28.284 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.124126.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.112130.log.gz 2023-10-11T11:23:28.284 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.129510.log 2023-10-11T11:23:28.295 INFO:teuthology.orchestra.run.smithi111.stderr: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.124126.log.gz 2023-10-11T11:23:28.295 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.129510.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.129510.log.gz 2023-10-11T11:23:28.295 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.127654.log 2023-10-11T11:23:28.295 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.127654.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.127654.log.gz 2023-10-11T11:23:28.296 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121203.log 2023-10-11T11:23:28.296 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.121203.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.121203.log.gz 2023-10-11T11:23:28.296 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119940.log 2023-10-11T11:23:28.296 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.119940.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.119940.log.gz 2023-10-11T11:23:28.296 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.135994.log 2023-10-11T11:23:28.296 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.135994.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.135994.log.gz 2023-10-11T11:23:28.297 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138075.log 2023-10-11T11:23:28.297 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.138075.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.138075.log.gz 2023-10-11T11:23:28.297 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126253.log 2023-10-11T11:23:28.306 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.126253.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.126253.log.gz 2023-10-11T11:23:28.306 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.146663.log 2023-10-11T11:23:28.306 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.admin.146663.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.146663.log.gz 2023-10-11T11:23:28.306 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.100601.log 2023-10-11T11:23:28.306 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.admin.100601.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.100601.log.gz 2023-10-11T11:23:28.307 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.110002.log 2023-10-11T11:23:28.307 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.110002.log: 81.7% -- replaced with /var/log/ceph/cluster2-client.mirror.110002.log.gz 2023-10-11T11:23:28.307 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.128453.log 2023-10-11T11:23:28.307 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.128453.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.128453.log.gz 2023-10-11T11:23:28.307 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.110313.log 2023-10-11T11:23:28.308 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.110313.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.110313.log.gz 2023-10-11T11:23:28.308 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.127954.log 2023-10-11T11:23:28.308 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.127954.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.127954.log.gz 2023-10-11T11:23:28.308 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127133.log 2023-10-11T11:23:28.308 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.127133.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.127133.log.gz 2023-10-11T11:23:28.309 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119923.log 2023-10-11T11:23:28.309 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.119923.log: 80.3% -- replaced with /var/log/ceph/cluster2-client.mirror.119923.log.gz 2023-10-11T11:23:28.316 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138345.log 2023-10-11T11:23:28.316 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.138345.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.138345.log.gz 2023-10-11T11:23:28.316 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.145419.log 2023-10-11T11:23:28.328 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.admin.145419.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.145419.log.gz 2023-10-11T11:23:28.328 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.129994.log 2023-10-11T11:23:28.339 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.129994.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.129994.log.gz 2023-10-11T11:23:28.339 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.133152.log 2023-10-11T11:23:28.351 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.133152.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.133152.log.gz 2023-10-11T11:23:28.362 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108802.log 2023-10-11T11:23:28.368 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.108802.log: 87.1% -- replaced with /var/log/ceph/cluster1-client.mirror.108802.log.gz 2023-10-11T11:23:28.368 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.115682.log 2023-10-11T11:23:28.387 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.115682.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.115682.log.gz 2023-10-11T11:23:28.387 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126294.log 2023-10-11T11:23:28.393 INFO:teuthology.orchestra.run.smithi111.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.117278.log.gz 2023-10-11T11:23:28.395 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.126294.log: 91.8% -- replaced with /var/log/ceph/cluster1-client.mirror.126294.log.gz 2023-10-11T11:23:28.396 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.128295.log 2023-10-11T11:23:28.397 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.128295.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.128295.log.gz 2023-10-11T11:23:28.397 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.3.111161.log 2023-10-11T11:23:28.400 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.3.111161.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.134031.log 2023-10-11T11:23:28.407 INFO:teuthology.orchestra.run.smithi111.stderr: 93.7% -- replaced with /var/log/ceph/cluster1-client.mirror.3.111161.log.gz 2023-10-11T11:23:28.409 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.134031.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.134031.log.gz 2023-10-11T11:23:28.409 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.141556.log 2023-10-11T11:23:28.409 INFO:teuthology.orchestra.run.smithi111.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.110178.log.gz 2023-10-11T11:23:28.411 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125023.log 2023-10-11T11:23:28.411 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.141556.log: /var/log/ceph/cluster1-client.mirror.125023.log: 91.3% -- replaced with /var/log/ceph/cluster2-client.mirror.141556.log.gz 2023-10-11T11:23:28.412 INFO:teuthology.orchestra.run.smithi111.stderr: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.125023.log.gz 2023-10-11T11:23:28.412 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117937.log 2023-10-11T11:23:28.413 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.131614.log 2023-10-11T11:23:28.413 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.117937.log: 80.3% -- replaced with /var/log/ceph/cluster2-client.mirror.117937.log.gz 2023-10-11T11:23:28.413 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.131614.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.118233.log.gz 2023-10-11T11:23:28.415 INFO:teuthology.orchestra.run.smithi111.stderr: 92.3% -- replaced with /var/log/ceph/cluster2-client.mirror.131614.log.gz 2023-10-11T11:23:28.415 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.114323.log 2023-10-11T11:23:28.416 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.134893.log 2023-10-11T11:23:28.416 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.114323.log: /var/log/ceph/cluster2-client.mirror.134893.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.114323.log.gz 2023-10-11T11:23:28.418 INFO:teuthology.orchestra.run.smithi111.stderr: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.134893.log.gz 2023-10-11T11:23:28.418 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.120879.log 2023-10-11T11:23:28.419 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114260.log 2023-10-11T11:23:28.419 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.120879.log: /var/log/ceph/cluster1-client.mirror.114260.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.120879.log.gz 2023-10-11T11:23:28.424 INFO:teuthology.orchestra.run.smithi111.stderr: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.114260.log.gz 2023-10-11T11:23:28.424 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.142764.log 2023-10-11T11:23:28.425 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.141700.log 2023-10-11T11:23:28.425 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.142764.log: /var/log/ceph/cluster2-client.mirror.141700.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.142764.log.gz 2023-10-11T11:23:28.427 INFO:teuthology.orchestra.run.smithi111.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.141700.log.gz 2023-10-11T11:23:28.428 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127561.log 2023-10-11T11:23:28.428 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116944.log 2023-10-11T11:23:28.428 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.127561.log: /var/log/ceph/cluster2-client.mirror.116944.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.127561.log.gz 2023-10-11T11:23:28.429 INFO:teuthology.orchestra.run.smithi111.stderr: 80.4% -- replaced with /var/log/ceph/cluster2-client.mirror.116944.log.gz 2023-10-11T11:23:28.430 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.138669.log 2023-10-11T11:23:28.430 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118430.log 2023-10-11T11:23:28.431 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.138669.log: /var/log/ceph/cluster2-client.mirror.118430.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.138669.log.gz 2023-10-11T11:23:28.431 INFO:teuthology.orchestra.run.smithi111.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.118430.log.gz 2023-10-11T11:23:28.431 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.128497.log 2023-10-11T11:23:28.431 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111636.log 2023-10-11T11:23:28.432 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.128497.log: /var/log/ceph/cluster1-client.mirror.111636.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.128497.log.gz 2023-10-11T11:23:28.435 INFO:teuthology.orchestra.run.smithi111.stderr: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.111636.log.gz 2023-10-11T11:23:28.435 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.112736.log 2023-10-11T11:23:28.436 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.133760.log 2023-10-11T11:23:28.436 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.112736.log: /var/log/ceph/cluster2-client.mirror.133760.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.112736.log.gz 2023-10-11T11:23:28.438 INFO:teuthology.orchestra.run.smithi111.stderr: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.133760.log.gz 2023-10-11T11:23:28.438 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.134279.log 2023-10-11T11:23:28.439 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122797.log 2023-10-11T11:23:28.440 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.134279.log: /var/log/ceph/cluster1-client.mirror.122797.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.122797.log.gz 2023-10-11T11:23:28.440 INFO:teuthology.orchestra.run.smithi111.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.134279.log.gz 2023-10-11T11:23:28.440 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.107409.log 2023-10-11T11:23:28.441 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117523.log 2023-10-11T11:23:28.441 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.admin.107409.log: 79.9% -- replaced with /var/log/ceph/cluster2-client.admin.107409.log.gz 2023-10-11T11:23:28.451 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.117523.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.138634.log 2023-10-11T11:23:28.452 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119903.log 2023-10-11T11:23:28.452 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.138634.log: 2.6% -- replaced with /var/log/ceph/cluster2-client.mirror.138634.log.gz 2023-10-11T11:23:28.462 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.119903.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.114772.log 2023-10-11T11:23:28.464 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.114772.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.114772.log.gz 2023-10-11T11:23:28.465 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116339.log 2023-10-11T11:23:28.466 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.116339.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.116339.log.gz 2023-10-11T11:23:28.466 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126229.log 2023-10-11T11:23:28.467 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.126229.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.126229.log.gz 2023-10-11T11:23:28.467 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.121058.log 2023-10-11T11:23:28.468 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.121058.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.121058.log.gz 2023-10-11T11:23:28.469 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124910.log 2023-10-11T11:23:28.469 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.124910.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.124910.log.gz 2023-10-11T11:23:28.470 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.126768.log 2023-10-11T11:23:28.471 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.126768.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.126768.log.gz 2023-10-11T11:23:28.471 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111472.log 2023-10-11T11:23:28.472 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.111472.log: 88.7% -- replaced with /var/log/ceph/cluster1-client.mirror.111472.log.gz 2023-10-11T11:23:28.472 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.139945.log 2023-10-11T11:23:28.473 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.139945.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.139945.log.gz 2023-10-11T11:23:28.474 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.133256.log 2023-10-11T11:23:28.477 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.133256.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.133256.log.gz 2023-10-11T11:23:28.477 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112582.log 2023-10-11T11:23:28.478 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.112582.log: 89.2% -- replaced with /var/log/ceph/cluster1-client.mirror.112582.log.gz 2023-10-11T11:23:28.478 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117323.log 2023-10-11T11:23:28.479 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.117323.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.117323.log.gz 2023-10-11T11:23:28.480 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116260.log 2023-10-11T11:23:28.481 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.116260.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.116260.log.gz 2023-10-11T11:23:28.481 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119231.log 2023-10-11T11:23:28.481 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.119231.log: 80.4% -- replaced with /var/log/ceph/cluster2-client.mirror.119231.log.gz 2023-10-11T11:23:28.482 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.110258.log 2023-10-11T11:23:28.484 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.110258.log: 91.4% -- replaced with /var/log/ceph/cluster2-client.mirror.110258.log.gz 2023-10-11T11:23:28.484 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.143663.log 2023-10-11T11:23:28.484 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.admin.143663.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.143663.log.gz 2023-10-11T11:23:28.485 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.107563.log 2023-10-11T11:23:28.486 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.107563.log: 54.5% -- replaced with /var/log/ceph/cluster2-client.mirror.107563.log.gz 2023-10-11T11:23:28.486 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.114472.log 2023-10-11T11:23:28.487 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.114472.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.114472.log.gz 2023-10-11T11:23:28.487 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125269.log 2023-10-11T11:23:28.488 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.125269.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.125269.log.gz 2023-10-11T11:23:28.489 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.122045.log 2023-10-11T11:23:28.489 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.122045.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.122045.log.gz 2023-10-11T11:23:28.490 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.112788.log 2023-10-11T11:23:28.490 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.admin.112788.log: 88.3% -- replaced with /var/log/ceph/cluster1-client.admin.112788.log.gz 2023-10-11T11:23:28.491 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.131018.log 2023-10-11T11:23:28.492 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.131018.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.131018.log.gz 2023-10-11T11:23:28.492 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.113212.log 2023-10-11T11:23:28.493 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.113212.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.113212.log.gz 2023-10-11T11:23:28.493 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.113103.log 2023-10-11T11:23:28.494 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.113103.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112943.log 2023-10-11T11:23:28.501 INFO:teuthology.orchestra.run.smithi111.stderr: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.113103.log.gz 2023-10-11T11:23:28.501 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.112943.log: 89.3% -- replaced with /var/log/ceph/cluster1-client.mirror.112943.log.gz 2023-10-11T11:23:28.501 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119686.log 2023-10-11T11:23:28.502 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.119686.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.119686.log.gz 2023-10-11T11:23:28.502 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.134176.log 2023-10-11T11:23:28.503 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.134176.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.134176.log.gz 2023-10-11T11:23:28.503 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.105716.log 2023-10-11T11:23:28.504 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.admin.105716.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.105716.log.gz 2023-10-11T11:23:28.505 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.132197.log 2023-10-11T11:23:28.506 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.132197.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.132197.log.gz 2023-10-11T11:23:28.507 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.133380.log 2023-10-11T11:23:28.508 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.133380.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.114017.log 2023-10-11T11:23:28.509 INFO:teuthology.orchestra.run.smithi111.stderr: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.133380.log.gz 2023-10-11T11:23:28.511 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.114017.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.117191.log.gz 2023-10-11T11:23:28.511 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120995.log 2023-10-11T11:23:28.512 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.120995.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.120995.log.gz 2023-10-11T11:23:28.512 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.130605.log 2023-10-11T11:23:28.514 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.130605.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.130605.log.gz 2023-10-11T11:23:28.514 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121036.log 2023-10-11T11:23:28.515 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.121036.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.114017.log.gz 2023-10-11T11:23:28.515 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.135168.log 2023-10-11T11:23:28.516 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.135168.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.135168.log.gz 2023-10-11T11:23:28.516 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 92.2%--verbose -- -- replaced with /var/log/ceph/cluster1-client.mirror.121036.log.gz/var/log/ceph/cluster2-client.mirror.134658.log 2023-10-11T11:23:28.517 INFO:teuthology.orchestra.run.smithi111.stderr: 2023-10-11T11:23:28.518 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118053.log 2023-10-11T11:23:28.519 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.134658.log: /var/log/ceph/cluster2-client.mirror.118053.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.134658.log.gz 2023-10-11T11:23:28.519 INFO:teuthology.orchestra.run.smithi111.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.118053.log.gz 2023-10-11T11:23:28.519 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.106680.log 2023-10-11T11:23:28.520 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121950.log 2023-10-11T11:23:28.520 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.106680.log: /var/log/ceph/cluster1-client.mirror.121950.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.121950.log.gz 2023-10-11T11:23:28.521 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.122194.log 2023-10-11T11:23:28.524 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.122194.log: 92.3% -- replaced with /var/log/ceph/cluster2-client.mirror.122194.log.gz 2023-10-11T11:23:28.524 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.135726.log 2023-10-11T11:23:28.524 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.135726.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.135726.log.gz 2023-10-11T11:23:28.525 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136410.log 2023-10-11T11:23:28.531 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.136410.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.136410.log.gz 2023-10-11T11:23:28.531 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.110202.log 2023-10-11T11:23:28.531 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.admin.110202.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.110202.log.gz 2023-10-11T11:23:28.531 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.108360.log 2023-10-11T11:23:28.532 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.admin.108360.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.108360.log.gz 2023-10-11T11:23:28.532 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119528.log 2023-10-11T11:23:28.533 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.119528.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.119528.log.gz 2023-10-11T11:23:28.534 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137018.log 2023-10-11T11:23:28.534 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.137018.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.137018.log.gz 2023-10-11T11:23:28.535 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.141807.log 2023-10-11T11:23:28.537 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.141807.log: 91.4% -- replaced with /var/log/ceph/cluster2-client.mirror.141807.log.gz 2023-10-11T11:23:28.537 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.129284.log 2023-10-11T11:23:28.538 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.129284.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.129284.log.gz 2023-10-11T11:23:28.538 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113994.log 2023-10-11T11:23:28.544 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.113994.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.113994.log.gz 2023-10-11T11:23:28.544 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.142608.log 2023-10-11T11:23:28.545 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.142608.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.142608.log.gz 2023-10-11T11:23:28.545 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123749.log 2023-10-11T11:23:28.546 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.123749.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.123749.log.gz 2023-10-11T11:23:28.546 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.128021.log 2023-10-11T11:23:28.547 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.128021.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.128021.log.gz 2023-10-11T11:23:28.548 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.143125.log 2023-10-11T11:23:28.548 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.143125.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.143125.log.gz 2023-10-11T11:23:28.549 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119982.log 2023-10-11T11:23:28.554 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.119982.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.141535.log 2023-10-11T11:23:28.583 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.141535.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.141535.log.gz 2023-10-11T11:23:28.583 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138030.log 2023-10-11T11:23:28.583 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.138030.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.138030.log.gz 2023-10-11T11:23:28.583 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113039.log 2023-10-11T11:23:28.594 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.113039.log: 88.3% -- replaced with /var/log/ceph/cluster1-client.mirror.113039.log.gz 2023-10-11T11:23:28.594 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126746.log 2023-10-11T11:23:28.605 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.126746.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.145143.log 2023-10-11T11:23:28.605 INFO:teuthology.orchestra.run.smithi111.stderr: 92.1% -- replaced with /var/log/ceph/cluster1-client.mirror.126746.log.gz 2023-10-11T11:23:28.617 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.admin.145143.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.145143.log.gz 2023-10-11T11:23:28.618 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.129627.log 2023-10-11T11:23:28.632 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.129627.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110945.log 2023-10-11T11:23:28.632 INFO:teuthology.orchestra.run.smithi111.stderr: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.129627.log.gz 2023-10-11T11:23:28.632 INFO:teuthology.orchestra.run.smithi111.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.116282.log.gz 2023-10-11T11:23:28.639 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.110945.log: 88.5% -- replaced with /var/log/ceph/cluster1-client.mirror.110945.log.gz 2023-10-11T11:23:28.639 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113951.log 2023-10-11T11:23:28.644 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.113951.log: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.113951.log.gz 2023-10-11T11:23:28.645 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116774.log 2023-10-11T11:23:28.649 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.116774.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107296.log 2023-10-11T11:23:28.671 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.107296.log: 56.7% -- replaced with /var/log/ceph/cluster1-client.mirror.107296.log.gz 2023-10-11T11:23:28.671 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.3.119014.log 2023-10-11T11:23:28.674 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.3.119014.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.106498.log 2023-10-11T11:23:28.674 INFO:teuthology.orchestra.run.smithi111.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.117523.log.gz 2023-10-11T11:23:28.674 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.admin.106498.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.106498.log.gz 2023-10-11T11:23:28.674 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.125697.log 2023-10-11T11:23:28.685 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.125697.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.125697.log.gz 2023-10-11T11:23:28.685 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113541.log 2023-10-11T11:23:28.685 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.113541.log: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.113541.log.gz 2023-10-11T11:23:28.685 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116319.log 2023-10-11T11:23:28.686 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.116319.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.116319.log.gz 2023-10-11T11:23:28.686 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.135044.log 2023-10-11T11:23:28.686 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.135044.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.135044.log.gz 2023-10-11T11:23:28.686 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110897.log 2023-10-11T11:23:28.686 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.110897.log: 88.5% -- replaced with /var/log/ceph/cluster1-client.mirror.110897.log.gz 2023-10-11T11:23:28.687 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.142281.log 2023-10-11T11:23:28.696 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.142281.log: 91.2% -- replaced with /var/log/ceph/cluster2-client.mirror.142281.log.gz 2023-10-11T11:23:28.696 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.122658.log 2023-10-11T11:23:28.696 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.122658.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.122658.log.gz 2023-10-11T11:23:28.696 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.142261.log 2023-10-11T11:23:28.697 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.142261.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.142261.log.gz 2023-10-11T11:23:28.697 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.115039.log 2023-10-11T11:23:28.697 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.115039.log: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.106680.log.gz 2023-10-11T11:23:28.700 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.108748.log 2023-10-11T11:23:28.722 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.108748.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114037.log 2023-10-11T11:23:28.723 INFO:teuthology.orchestra.run.smithi111.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.116774.log.gz 2023-10-11T11:23:28.723 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.114037.log: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.114037.log.gz 2023-10-11T11:23:28.723 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.115859.log 2023-10-11T11:23:28.723 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.115859.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.115859.log.gz 2023-10-11T11:23:28.724 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.143100.log 2023-10-11T11:23:28.724 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.143100.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.143100.log.gz 2023-10-11T11:23:28.724 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.111862.log 2023-10-11T11:23:28.734 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.111862.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.111862.log.gz 2023-10-11T11:23:28.734 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113584.log 2023-10-11T11:23:28.734 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.113584.log: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.113584.log.gz 2023-10-11T11:23:28.734 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.134134.log 2023-10-11T11:23:28.735 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.134134.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.134134.log.gz 2023-10-11T11:23:28.735 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.134217.log 2023-10-11T11:23:28.735 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.134217.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.134217.log.gz 2023-10-11T11:23:28.735 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.109188.log 2023-10-11T11:23:28.744 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.109188.log: 93.3% -- replaced with /var/log/ceph/cluster2-client.mirror.109188.log.gz 2023-10-11T11:23:28.745 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124106.log 2023-10-11T11:23:28.745 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.124106.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.124106.log.gz 2023-10-11T11:23:28.745 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136613.log 2023-10-11T11:23:28.745 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.136613.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.136613.log.gz 2023-10-11T11:23:28.745 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.109230.log 2023-10-11T11:23:28.746 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.109230.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.109230.log.gz 2023-10-11T11:23:28.746 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.122985.log 2023-10-11T11:23:28.756 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.122985.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.122985.log.gz 2023-10-11T11:23:28.756 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.134052.log 2023-10-11T11:23:28.756 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.134052.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.134052.log.gz 2023-10-11T11:23:28.756 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.133277.log 2023-10-11T11:23:28.756 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.133277.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.133277.log.gz 2023-10-11T11:23:28.757 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118534.log 2023-10-11T11:23:28.757 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.118534.log: 80.2% -- replaced with /var/log/ceph/cluster2-client.mirror.118534.log.gz 2023-10-11T11:23:28.757 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122093.log 2023-10-11T11:23:28.757 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.122093.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.122093.log.gz 2023-10-11T11:23:28.758 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.113888.log 2023-10-11T11:23:28.758 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.113888.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.113888.log.gz 2023-10-11T11:23:28.758 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.133215.log 2023-10-11T11:23:28.766 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.133215.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.133215.log.gz 2023-10-11T11:23:28.767 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.141111.log 2023-10-11T11:23:28.780 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.141111.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.141111.log.gz 2023-10-11T11:23:28.781 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119211.log 2023-10-11T11:23:28.781 INFO:teuthology.orchestra.run.smithi111.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.119903.log.gz 2023-10-11T11:23:28.787 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.119211.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.139737.log 2023-10-11T11:23:28.794 INFO:teuthology.orchestra.run.smithi111.stderr: 94.0% -- replaced with /var/log/ceph/cluster2-client.mirror.3.119014.log.gz 2023-10-11T11:23:28.802 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.135250.log 2023-10-11T11:23:28.805 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.135250.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.135250.log.gz 2023-10-11T11:23:28.806 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109325.log 2023-10-11T11:23:28.806 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.109325.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.109325.log.gz 2023-10-11T11:23:28.807 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.120784.log 2023-10-11T11:23:28.809 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.120784.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138255.log 2023-10-11T11:23:28.810 INFO:teuthology.orchestra.run.smithi111.stderr: 92.3% -- replaced with /var/log/ceph/cluster2-client.mirror.120784.log.gz 2023-10-11T11:23:28.810 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.139737.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.139737.log.gz 2023-10-11T11:23:28.811 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.138255.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.138255.log.gz 2023-10-11T11:23:28.811 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.130813.log 2023-10-11T11:23:28.820 INFO:teuthology.orchestra.run.smithi111.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.119982.log.gz 2023-10-11T11:23:28.829 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.130813.log: 92.3% -- replaced with /var/log/ceph/cluster2-client.mirror.130813.log.gz 2023-10-11T11:23:28.829 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137468.log 2023-10-11T11:23:28.829 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.137468.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.137468.log.gz 2023-10-11T11:23:28.830 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.129380.log 2023-10-11T11:23:28.830 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.129380.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.129380.log.gz 2023-10-11T11:23:28.830 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109442.log 2023-10-11T11:23:28.831 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.109442.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.109442.log.gz 2023-10-11T11:23:28.831 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137918.log 2023-10-11T11:23:28.831 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.137918.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.137918.log.gz 2023-10-11T11:23:28.831 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.3.106755.log 2023-10-11T11:23:28.832 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.3.106755.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.3.106755.log.gz 2023-10-11T11:23:28.834 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.114612.log 2023-10-11T11:23:28.835 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.114612.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.114612.log.gz 2023-10-11T11:23:28.835 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.125362.log 2023-10-11T11:23:28.836 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.125362.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.125362.log.gz 2023-10-11T11:23:28.836 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111978.log 2023-10-11T11:23:28.840 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.111978.log: 89.0% -- replaced with /var/log/ceph/cluster1-client.mirror.111978.log.gz 2023-10-11T11:23:28.840 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137310.log 2023-10-11T11:23:28.841 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.137310.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.137310.log.gz 2023-10-11T11:23:28.841 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.111658.log 2023-10-11T11:23:28.841 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.111658.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.111658.log.gz 2023-10-11T11:23:28.841 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.128723.log 2023-10-11T11:23:28.841 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.128723.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.128723.log.gz 2023-10-11T11:23:28.842 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.136227.log 2023-10-11T11:23:28.843 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.136227.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.136227.log.gz 2023-10-11T11:23:28.843 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109124.log 2023-10-11T11:23:28.844 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.109124.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.109124.log.gz 2023-10-11T11:23:28.844 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.130744.log 2023-10-11T11:23:28.845 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.130744.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.130744.log.gz 2023-10-11T11:23:28.845 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.133006.log 2023-10-11T11:23:28.848 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.133006.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.133006.log.gz 2023-10-11T11:23:28.849 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.144491.log 2023-10-11T11:23:28.850 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.admin.144491.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.144491.log.gz 2023-10-11T11:23:28.850 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.143075.log 2023-10-11T11:23:28.850 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.143075.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.143075.log.gz 2023-10-11T11:23:28.850 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136905.log 2023-10-11T11:23:28.851 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.136905.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.136905.log.gz 2023-10-11T11:23:28.851 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.142474.log 2023-10-11T11:23:28.852 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.142474.log: 82.6% -- replaced with /var/log/ceph/cluster2-client.mirror.142474.log.gz 2023-10-11T11:23:28.852 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.105946.log 2023-10-11T11:23:28.853 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.admin.105946.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.105946.log.gz 2023-10-11T11:23:28.853 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.128045.log 2023-10-11T11:23:28.853 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.128045.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.128045.log.gz 2023-10-11T11:23:28.854 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.105578.log 2023-10-11T11:23:28.855 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.admin.105578.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.105578.log.gz 2023-10-11T11:23:28.855 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123797.log 2023-10-11T11:23:28.855 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.123797.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.123797.log.gz 2023-10-11T11:23:28.855 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119189.log 2023-10-11T11:23:28.856 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.119189.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.119189.log.gz 2023-10-11T11:23:28.856 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116900.log 2023-10-11T11:23:28.857 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.147215.log 2023-10-11T11:23:28.857 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.116900.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.116900.log.gz 2023-10-11T11:23:28.858 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.admin.147215.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.147215.log.gz 2023-10-11T11:23:28.858 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.115326.log 2023-10-11T11:23:28.866 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.142030.log 2023-10-11T11:23:28.866 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.115326.log: /var/log/ceph/cluster2-client.mirror.142030.log: 91.2% -- replaced with /var/log/ceph/cluster2-client.mirror.142030.log.gz 2023-10-11T11:23:28.867 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.129356.log 2023-10-11T11:23:28.867 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.129356.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.129356.log.gz 2023-10-11T11:23:28.868 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.106661.log 2023-10-11T11:23:28.868 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.0.106661.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.143433.log 2023-10-11T11:23:28.868 INFO:teuthology.orchestra.run.smithi111.stderr: 93.1% -- replaced with /var/log/ceph/cluster1-client.mirror.0.106661.log.gz 2023-10-11T11:23:28.869 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.admin.143433.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.143433.log.gz 2023-10-11T11:23:28.869 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125317.log 2023-10-11T11:23:28.869 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.125317.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.125317.log.gz 2023-10-11T11:23:28.870 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.1.132620.log 2023-10-11T11:23:28.872 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.1.132620.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.135126.log 2023-10-11T11:23:28.879 INFO:teuthology.orchestra.run.smithi111.stderr: 93.2% -- replaced with /var/log/ceph/cluster2-client.mirror.1.132620.log.gz 2023-10-11T11:23:28.882 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.135126.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.135126.log.gz 2023-10-11T11:23:28.882 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.1.106785.log 2023-10-11T11:23:28.887 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.1.106785.log: 93.1% -- replaced with /var/log/ceph/cluster2-client.mirror.1.106785.log.gz 2023-10-11T11:23:28.887 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127585.log 2023-10-11T11:23:28.888 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.127585.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.127585.log.gz 2023-10-11T11:23:28.888 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.107152.log 2023-10-11T11:23:28.890 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.admin.107152.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.107152.log.gz 2023-10-11T11:23:28.890 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.113802.log 2023-10-11T11:23:28.895 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.113802.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.113802.log.gz 2023-10-11T11:23:28.895 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.131083.log 2023-10-11T11:23:28.897 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.131083.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.131083.log.gz 2023-10-11T11:23:28.898 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.132313.log 2023-10-11T11:23:28.898 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.132313.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.132313.log.gz 2023-10-11T11:23:28.899 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124426.log 2023-10-11T11:23:28.900 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.124426.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.124426.log.gz 2023-10-11T11:23:28.900 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137040.log 2023-10-11T11:23:28.901 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.137040.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.137040.log.gz 2023-10-11T11:23:28.901 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.132376.log 2023-10-11T11:23:28.903 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.132376.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.132376.log.gz 2023-10-11T11:23:28.903 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119728.log 2023-10-11T11:23:28.907 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.119728.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.132467.log 2023-10-11T11:23:28.919 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.132467.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.132467.log.gz 2023-10-11T11:23:28.919 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126931.log 2023-10-11T11:23:28.931 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.126931.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.126931.log.gz 2023-10-11T11:23:28.931 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.106785.log 2023-10-11T11:23:28.942 INFO:teuthology.orchestra.run.smithi111.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.119211.log.gz 2023-10-11T11:23:28.953 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.106785.log: 80.7% -- replaced with /var/log/ceph/cluster1-client.mirror.106785.log.gz 2023-10-11T11:23:28.953 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.135953.log 2023-10-11T11:23:28.954 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.135953.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.135953.log.gz 2023-10-11T11:23:28.954 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120947.log 2023-10-11T11:23:28.954 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.120947.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.120947.log.gz 2023-10-11T11:23:28.955 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.128473.log 2023-10-11T11:23:28.955 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.128473.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.128473.log.gz 2023-10-11T11:23:28.955 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.130946.log 2023-10-11T11:23:28.956 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.130946.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.130946.log.gz 2023-10-11T11:23:28.956 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119706.log 2023-10-11T11:23:28.956 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.119706.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.108748.log.gz 2023-10-11T11:23:28.956 INFO:teuthology.orchestra.run.smithi111.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.119706.log.gz 2023-10-11T11:23:28.957 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.134447.log 2023-10-11T11:23:28.957 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.134447.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.134447.log.gz 2023-10-11T11:23:28.957 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.147031.log 2023-10-11T11:23:28.957 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.admin.147031.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.147031.log.gz 2023-10-11T11:23:28.958 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.132334.log 2023-10-11T11:23:28.960 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.132334.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.132334.log.gz 2023-10-11T11:23:28.961 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.120041.log 2023-10-11T11:23:28.961 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.120041.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.120041.log.gz 2023-10-11T11:23:28.961 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.144031.log 2023-10-11T11:23:28.961 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137715.log 2023-10-11T11:23:28.962 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.admin.144031.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.144031.log.gz 2023-10-11T11:23:28.962 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.137715.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.137715.log.gz 2023-10-11T11:23:28.962 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118875.log 2023-10-11T11:23:28.963 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117771.log 2023-10-11T11:23:28.963 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.118875.log: 80.3% -- replaced with /var/log/ceph/cluster2-client.mirror.118875.log.gz 2023-10-11T11:23:28.963 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.117771.log: 80.5% -- replaced with /var/log/ceph/cluster2-client.mirror.117771.log.gz 2023-10-11T11:23:28.963 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.122773.log 2023-10-11T11:23:28.963 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.122773.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.122773.log.gz 2023-10-11T11:23:28.963 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.135912.log 2023-10-11T11:23:28.964 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112714.log 2023-10-11T11:23:28.964 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.135912.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.135912.log.gz 2023-10-11T11:23:28.964 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.112714.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.114652.log 2023-10-11T11:23:28.965 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.114652.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.114652.log.gz 2023-10-11T11:23:28.967 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136995.log 2023-10-11T11:23:28.967 INFO:teuthology.orchestra.run.smithi111.stderr: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.112714.log.gz 2023-10-11T11:23:28.967 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125745.log 2023-10-11T11:23:28.968 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.136995.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.136995.log.gz 2023-10-11T11:23:28.968 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.125745.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.125745.log.gz 2023-10-11T11:23:28.968 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.110156.log 2023-10-11T11:23:28.969 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.110156.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.110156.log.gz 2023-10-11T11:23:28.969 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.131398.log 2023-10-11T11:23:28.970 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.131398.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.131398.log.gz 2023-10-11T11:23:28.970 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.107194.log 2023-10-11T11:23:28.971 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.admin.107194.log: 79.9% -- replaced with /var/log/ceph/cluster1-client.admin.107194.log.gz 2023-10-11T11:23:28.971 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119366.log 2023-10-11T11:23:28.981 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.119366.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.113392.log 2023-10-11T11:23:28.987 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.113392.log: 93.3% -- replaced with /var/log/ceph/cluster2-client.mirror.113392.log.gz 2023-10-11T11:23:28.992 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.134761.log 2023-10-11T11:23:28.992 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.134761.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.134761.log.gz 2023-10-11T11:23:28.992 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.139825.log 2023-10-11T11:23:28.993 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.139825.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.139825.log.gz 2023-10-11T11:23:28.993 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.139925.log 2023-10-11T11:23:28.993 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.139925.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.139925.log.gz 2023-10-11T11:23:28.994 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111930.log 2023-10-11T11:23:28.994 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.111930.log: 88.9% -- replaced with /var/log/ceph/cluster1-client.mirror.111930.log.gz 2023-10-11T11:23:28.994 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120291.log 2023-10-11T11:23:28.995 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.120291.log: 1.3% -- replaced with /var/log/ceph/cluster1-client.mirror.120291.log.gz 2023-10-11T11:23:28.995 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136568.log 2023-10-11T11:23:28.996 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.136568.log: 86.4% -- replaced with /var/log/ceph/cluster1-client.mirror.136568.log.gz 2023-10-11T11:23:28.996 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.131042.log 2023-10-11T11:23:28.997 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.131042.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.131042.log.gz 2023-10-11T11:23:28.997 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.113306.log 2023-10-11T11:23:29.000 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.113306.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.130196.log 2023-10-11T11:23:29.004 INFO:teuthology.orchestra.run.smithi111.stderr: 93.0% -- replaced with /var/log/ceph/cluster2-client.mirror.113306.log.gz 2023-10-11T11:23:29.005 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.130196.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.130196.log.gz 2023-10-11T11:23:29.005 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.115921.log 2023-10-11T11:23:29.006 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.115921.log: 80.6% -- replaced with /var/log/ceph/cluster2-client.mirror.115921.log.gz 2023-10-11T11:23:29.006 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123821.log 2023-10-11T11:23:29.007 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.123821.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.123821.log.gz 2023-10-11T11:23:29.007 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.115842.log 2023-10-11T11:23:29.008 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.115842.log: 80.4% -- replaced with /var/log/ceph/cluster2-client.mirror.115842.log.gz 2023-10-11T11:23:29.008 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116223.log 2023-10-11T11:23:29.009 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.116223.log: 80.3% -- replaced with /var/log/ceph/cluster2-client.mirror.116223.log.gz 2023-10-11T11:23:29.009 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.3.106832.log 2023-10-11T11:23:29.020 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.3.106832.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108704.log 2023-10-11T11:23:29.033 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.108704.log: 92.7% -- replaced with /var/log/ceph/cluster2-client.mirror.3.106832.log.gz 2023-10-11T11:23:29.037 INFO:teuthology.orchestra.run.smithi111.stderr: 93.0% -- replaced with /var/log/ceph/cluster1-client.mirror.108704.log.gz 2023-10-11T11:23:29.037 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119669.log 2023-10-11T11:23:29.038 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.119669.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118675.log 2023-10-11T11:23:29.038 INFO:teuthology.orchestra.run.smithi111.stderr: 80.4% -- replaced with /var/log/ceph/cluster2-client.mirror.119669.log.gz 2023-10-11T11:23:29.039 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.118675.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.118675.log.gz 2023-10-11T11:23:29.039 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118855.log 2023-10-11T11:23:29.042 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.118855.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.143221.log 2023-10-11T11:23:29.045 INFO:teuthology.orchestra.run.smithi111.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.115326.log.gz 2023-10-11T11:23:29.048 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.admin.143221.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.143221.log.gz 2023-10-11T11:23:29.052 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.125404.log 2023-10-11T11:23:29.053 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.125404.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.125404.log.gz 2023-10-11T11:23:29.053 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.135581.log 2023-10-11T11:23:29.055 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.135581.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.135581.log.gz 2023-10-11T11:23:29.055 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.145005.log 2023-10-11T11:23:29.056 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.admin.145005.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.145005.log.gz 2023-10-11T11:23:29.056 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.126726.log 2023-10-11T11:23:29.058 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.126726.log: 92.3% -- replaced with /var/log/ceph/cluster2-client.mirror.126726.log.gz 2023-10-11T11:23:29.058 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.108386.log 2023-10-11T11:23:29.059 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.admin.108386.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.108386.log.gz 2023-10-11T11:23:29.059 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.135188.log 2023-10-11T11:23:29.062 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.135188.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.135188.log.gz 2023-10-11T11:23:29.063 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.140179.log 2023-10-11T11:23:29.065 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.140179.log: 91.4% -- replaced with /var/log/ceph/cluster2-client.mirror.140179.log.gz 2023-10-11T11:23:29.065 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113412.log 2023-10-11T11:23:29.068 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.113412.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127633.log 2023-10-11T11:23:29.072 INFO:teuthology.orchestra.run.smithi111.stderr: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.113412.log.gz 2023-10-11T11:23:29.072 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.127633.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.127633.log.gz 2023-10-11T11:23:29.073 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.131924.log 2023-10-11T11:23:29.074 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.131924.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.131924.log.gz 2023-10-11T11:23:29.074 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.136207.log 2023-10-11T11:23:29.075 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.136207.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.136207.log.gz 2023-10-11T11:23:29.076 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.146755.log 2023-10-11T11:23:29.076 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.admin.146755.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.146755.log.gz 2023-10-11T11:23:29.076 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.145603.log 2023-10-11T11:23:29.077 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.admin.145603.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.145603.log.gz 2023-10-11T11:23:29.077 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.107512.log 2023-10-11T11:23:29.078 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.107512.log: 57.4% -- replaced with /var/log/ceph/cluster2-client.mirror.107512.log.gz 2023-10-11T11:23:29.078 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.125451.log 2023-10-11T11:23:29.080 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.125451.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.125451.log.gz 2023-10-11T11:23:29.080 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.1.114383.log 2023-10-11T11:23:29.083 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.1.114383.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123565.log 2023-10-11T11:23:29.097 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.123565.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.123565.log.gz 2023-10-11T11:23:29.097 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.133131.log 2023-10-11T11:23:29.108 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.133131.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.133990.log 2023-10-11T11:23:29.119 INFO:teuthology.orchestra.run.smithi111.stderr: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.133131.log.gz 2023-10-11T11:23:29.119 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.133990.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.133990.log.gz 2023-10-11T11:23:29.120 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.125003.log 2023-10-11T11:23:29.132 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.125003.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.130970.log 2023-10-11T11:23:29.143 INFO:teuthology.orchestra.run.smithi111.stderr: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.125003.log.gz 2023-10-11T11:23:29.155 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.130970.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.130970.log.gz 2023-10-11T11:23:29.156 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.144123.log 2023-10-11T11:23:29.165 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.admin.144123.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.144123.log.gz 2023-10-11T11:23:29.165 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.145695.log 2023-10-11T11:23:29.176 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.admin.145695.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.145695.log.gz 2023-10-11T11:23:29.176 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123398.log 2023-10-11T11:23:29.187 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.123398.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.134637.log 2023-10-11T11:23:29.187 INFO:teuthology.orchestra.run.smithi111.stderr: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.123398.log.gz 2023-10-11T11:23:29.198 INFO:teuthology.orchestra.run.smithi111.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.115039.log.gz 2023-10-11T11:23:29.198 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.134637.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136433.log 2023-10-11T11:23:29.198 INFO:teuthology.orchestra.run.smithi111.stderr: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.134637.log.gz 2023-10-11T11:23:29.200 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.136433.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.136433.log.gz 2023-10-11T11:23:29.200 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.141132.log 2023-10-11T11:23:29.200 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.141132.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.141132.log.gz 2023-10-11T11:23:29.201 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.131737.log 2023-10-11T11:23:29.202 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.131737.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.131737.log.gz 2023-10-11T11:23:29.202 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119511.log 2023-10-11T11:23:29.203 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.119511.log: 80.3% -- replaced with /var/log/ceph/cluster2-client.mirror.119511.log.gz 2023-10-11T11:23:29.203 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.113145.log 2023-10-11T11:23:29.204 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.113145.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.113145.log.gz 2023-10-11T11:23:29.205 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114088.log 2023-10-11T11:23:29.211 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.114088.log: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.114088.log.gz 2023-10-11T11:23:29.211 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.133969.log 2023-10-11T11:23:29.212 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.133969.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.133969.log.gz 2023-10-11T11:23:29.212 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116922.log 2023-10-11T11:23:29.216 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.116922.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.119366.log.gz 2023-10-11T11:23:29.216 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127157.log 2023-10-11T11:23:29.228 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.127157.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.127157.log.gz 2023-10-11T11:23:29.229 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.141855.log 2023-10-11T11:23:29.231 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.0.141855.log: 92.0% -- replaced with /var/log/ceph/cluster1-client.mirror.0.141855.log.gz 2023-10-11T11:23:29.231 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.146571.log 2023-10-11T11:23:29.231 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.admin.146571.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.146571.log.gz 2023-10-11T11:23:29.232 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.146617.log 2023-10-11T11:23:29.232 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.admin.146617.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.146617.log.gz 2023-10-11T11:23:29.233 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118351.log 2023-10-11T11:23:29.234 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.118351.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.118351.log.gz 2023-10-11T11:23:29.234 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118912.log 2023-10-11T11:23:29.239 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.118912.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.118912.log.gz 2023-10-11T11:23:29.239 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.135498.log 2023-10-11T11:23:29.240 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.135498.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.135498.log.gz 2023-10-11T11:23:29.240 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.125814.log 2023-10-11T11:23:29.241 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.125814.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.125814.log.gz 2023-10-11T11:23:29.242 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.129970.log 2023-10-11T11:23:29.242 INFO:teuthology.orchestra.run.smithi111.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.119728.log.gz 2023-10-11T11:23:29.243 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.129970.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.129970.log.gz 2023-10-11T11:23:29.243 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.142175.log 2023-10-11T11:23:29.244 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138008.log 2023-10-11T11:23:29.245 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.142175.log: /var/log/ceph/cluster1-client.mirror.138008.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.138008.log.gz 2023-10-11T11:23:29.245 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.131991.log 2023-10-11T11:23:29.245 INFO:teuthology.orchestra.run.smithi111.stderr: 91.3% -- replaced with /var/log/ceph/cluster2-client.mirror.142175.log.gz 2023-10-11T11:23:29.246 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.131991.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.131991.log.gz 2023-10-11T11:23:29.247 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.121541.log 2023-10-11T11:23:29.257 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.1.121541.log: 94.3% -- replaced with /var/log/ceph/cluster2-client.mirror.1.114383.log.gz 2023-10-11T11:23:29.258 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108583.log 2023-10-11T11:23:29.259 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125608.log 2023-10-11T11:23:29.260 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.108583.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.108583.log.gz 2023-10-11T11:23:29.261 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.125608.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.125608.log.gz 2023-10-11T11:23:29.262 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.133339.log 2023-10-11T11:23:29.262 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.133339.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.133339.log.gz 2023-10-11T11:23:29.263 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138548.log 2023-10-11T11:23:29.267 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127787.log 2023-10-11T11:23:29.268 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.138548.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.138548.log.gz 2023-10-11T11:23:29.268 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.127787.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.127787.log.gz 2023-10-11T11:23:29.268 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.146387.log 2023-10-11T11:23:29.268 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.105715.log 2023-10-11T11:23:29.270 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.admin.146387.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.146387.log.gz 2023-10-11T11:23:29.270 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.admin.105715.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.105715.log.gz 2023-10-11T11:23:29.270 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.123937.log 2023-10-11T11:23:29.270 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.123937.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.123937.log.gz 2023-10-11T11:23:29.271 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.105440.log 2023-10-11T11:23:29.271 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.admin.105440.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.105440.log.gz 2023-10-11T11:23:29.271 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.107478.log 2023-10-11T11:23:29.271 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.122309.log 2023-10-11T11:23:29.272 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.107478.log: 4.7% -- replaced with /var/log/ceph/cluster2-client.mirror.107478.log.gz 2023-10-11T11:23:29.273 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.122309.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.122309.log.gz 2023-10-11T11:23:29.273 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119169.log 2023-10-11T11:23:29.273 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.115525.log 2023-10-11T11:23:29.274 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.119169.log: /var/log/ceph/cluster2-client.mirror.115525.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.119169.log.gz 2023-10-11T11:23:29.276 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.135850.log 2023-10-11T11:23:29.279 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.135850.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.135850.log.gz 2023-10-11T11:23:29.279 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.130583.log 2023-10-11T11:23:29.290 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.130583.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.130583.log.gz 2023-10-11T11:23:29.290 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125947.log 2023-10-11T11:23:29.291 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.125947.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.125947.log.gz 2023-10-11T11:23:29.291 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.129669.log 2023-10-11T11:23:29.293 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.129669.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.129669.log.gz 2023-10-11T11:23:29.293 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.131805.log 2023-10-11T11:23:29.294 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.131805.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.131805.log.gz 2023-10-11T11:23:29.294 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119649.log 2023-10-11T11:23:29.302 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.119649.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116302.log 2023-10-11T11:23:29.325 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.116302.log: 80.2% -- replaced with /var/log/ceph/cluster2-client.mirror.116302.log.gz 2023-10-11T11:23:29.326 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.134093.log 2023-10-11T11:23:29.327 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.134093.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.110853.log 2023-10-11T11:23:29.327 INFO:teuthology.orchestra.run.smithi111.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.134093.log.gz 2023-10-11T11:23:29.341 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.110853.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.143387.log 2023-10-11T11:23:29.343 INFO:teuthology.orchestra.run.smithi111.stderr: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.110853.log.gz 2023-10-11T11:23:29.355 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.133318.log 2023-10-11T11:23:29.366 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.admin.143387.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.143387.log.gz 2023-10-11T11:23:29.377 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.133318.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110603.log 2023-10-11T11:23:29.378 INFO:teuthology.orchestra.run.smithi111.stderr: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.133318.log.gz 2023-10-11T11:23:29.388 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.110603.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122357.log 2023-10-11T11:23:29.389 INFO:teuthology.orchestra.run.smithi111.stderr: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.110603.log.gz 2023-10-11T11:23:29.400 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.122357.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.122357.log.gz 2023-10-11T11:23:29.400 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.146019.log 2023-10-11T11:23:29.417 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.admin.146019.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.146019.log.gz 2023-10-11T11:23:29.417 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.121541.log 2023-10-11T11:23:29.428 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.121541.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.127085.log 2023-10-11T11:23:29.443 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.127085.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.127085.log.gz 2023-10-11T11:23:29.443 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.113845.log 2023-10-11T11:23:29.451 INFO:teuthology.orchestra.run.smithi111.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.118855.log.gz 2023-10-11T11:23:29.452 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.113845.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127867.log 2023-10-11T11:23:29.452 INFO:teuthology.orchestra.run.smithi111.stderr: 93.3% -- replaced with /var/log/ceph/cluster2-client.mirror.113845.log.gz 2023-10-11T11:23:29.452 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.127867.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.127867.log.gz 2023-10-11T11:23:29.453 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.122521.log 2023-10-11T11:23:29.454 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.122521.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.122521.log.gz 2023-10-11T11:23:29.454 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.128545.log 2023-10-11T11:23:29.455 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.128545.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.128545.log.gz 2023-10-11T11:23:29.455 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.130268.log 2023-10-11T11:23:29.456 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.130268.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.130268.log.gz 2023-10-11T11:23:29.457 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113822.log 2023-10-11T11:23:29.462 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.113822.log: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.113822.log.gz 2023-10-11T11:23:29.463 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118371.log 2023-10-11T11:23:29.475 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.118371.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.118371.log.gz 2023-10-11T11:23:29.475 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.126137.log 2023-10-11T11:23:29.492 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.126137.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.126137.log.gz 2023-10-11T11:23:29.492 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.115752.log 2023-10-11T11:23:29.503 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.115752.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.115752.log.gz 2023-10-11T11:23:29.504 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.132758.log 2023-10-11T11:23:29.512 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.132758.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.3.120103.log 2023-10-11T11:23:29.512 INFO:teuthology.orchestra.run.smithi111.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.132758.log.gz 2023-10-11T11:23:29.523 INFO:teuthology.orchestra.run.smithi111.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.119649.log.gz 2023-10-11T11:23:29.523 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.3.120103.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.120387.log 2023-10-11T11:23:29.534 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.120387.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.120387.log.gz 2023-10-11T11:23:29.534 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.140415.log 2023-10-11T11:23:29.545 INFO:teuthology.orchestra.run.smithi111.stderr: 93.2% -- replaced with /var/log/ceph/cluster1-client.mirror.3.120103.log.gz 2023-10-11T11:23:29.545 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.140415.log: 80.4% -- replaced with /var/log/ceph/cluster2-client.mirror.140415.log.gz 2023-10-11T11:23:29.545 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.146709.log 2023-10-11T11:23:29.545 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.admin.146709.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.146709.log.gz 2023-10-11T11:23:29.546 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.122236.log 2023-10-11T11:23:29.546 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.122236.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.122236.log.gz 2023-10-11T11:23:29.546 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.147169.log 2023-10-11T11:23:29.546 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.admin.147169.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.147169.log.gz 2023-10-11T11:23:29.547 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121909.log 2023-10-11T11:23:29.547 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.121909.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.121909.log.gz 2023-10-11T11:23:29.547 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119881.log 2023-10-11T11:23:29.547 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.119881.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.119881.log.gz 2023-10-11T11:23:29.548 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117464.log 2023-10-11T11:23:29.548 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.117464.log: 80.2% -- replaced with /var/log/ceph/cluster2-client.mirror.117464.log.gz 2023-10-11T11:23:29.548 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.142881.log 2023-10-11T11:23:29.549 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.142881.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.142881.log.gz 2023-10-11T11:23:29.549 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136973.log 2023-10-11T11:23:29.549 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.136973.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.136973.log.gz 2023-10-11T11:23:29.550 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127383.log 2023-10-11T11:23:29.556 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.127383.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.127383.log.gz 2023-10-11T11:23:29.556 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.107347.log 2023-10-11T11:23:29.557 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.admin.107347.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.107347.log.gz 2023-10-11T11:23:29.557 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.146341.log 2023-10-11T11:23:29.558 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.admin.146341.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.146341.log.gz 2023-10-11T11:23:29.558 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.123473.log 2023-10-11T11:23:29.559 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.123473.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.123473.log.gz 2023-10-11T11:23:29.560 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.133463.log 2023-10-11T11:23:29.560 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.133463.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.133463.log.gz 2023-10-11T11:23:29.560 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121820.log 2023-10-11T11:23:29.561 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.121820.log: 87.1% -- replaced with /var/log/ceph/cluster1-client.mirror.121820.log.gz 2023-10-11T11:23:29.561 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.139908.log 2023-10-11T11:23:29.561 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.139908.log: 80.5% -- replaced with /var/log/ceph/cluster2-client.mirror.139908.log.gz 2023-10-11T11:23:29.562 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124506.log 2023-10-11T11:23:29.562 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.124506.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.124506.log.gz 2023-10-11T11:23:29.563 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127891.log 2023-10-11T11:23:29.563 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.127891.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.127891.log.gz 2023-10-11T11:23:29.563 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.128180.log 2023-10-11T11:23:29.564 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.128180.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.128180.log.gz 2023-10-11T11:23:29.564 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.133802.log 2023-10-11T11:23:29.567 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.133802.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.133802.log.gz 2023-10-11T11:23:29.567 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.106314.log 2023-10-11T11:23:29.567 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.admin.106314.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.106314.log.gz 2023-10-11T11:23:29.568 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125341.log 2023-10-11T11:23:29.568 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.125341.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.125341.log.gz 2023-10-11T11:23:29.569 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.114900.log 2023-10-11T11:23:29.569 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.114900.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.114900.log.gz 2023-10-11T11:23:29.569 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.142361.log 2023-10-11T11:23:29.572 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.142361.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117996.log 2023-10-11T11:23:29.582 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.117996.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122381.log 2023-10-11T11:23:29.587 INFO:teuthology.orchestra.run.smithi111.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.115525.log.gz 2023-10-11T11:23:29.587 INFO:teuthology.orchestra.run.smithi111.stderr: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.121541.log.gz 2023-10-11T11:23:29.589 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.122381.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.122381.log.gz 2023-10-11T11:23:29.595 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.112168.log 2023-10-11T11:23:29.596 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.admin.112168.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.admin.112168.log.gz 2023-10-11T11:23:29.606 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.135808.log 2023-10-11T11:23:29.609 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.135808.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.135808.log.gz 2023-10-11T11:23:29.610 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.95454.log 2023-10-11T11:23:29.610 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph.tmp-client.admin.95454.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.95454.log.gz 2023-10-11T11:23:29.611 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.115980.log 2023-10-11T11:23:29.621 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.115980.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126455.log 2023-10-11T11:23:29.636 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.126455.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.126455.log.gz 2023-10-11T11:23:29.637 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.121131.log 2023-10-11T11:23:29.647 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.121131.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.121131.log.gz 2023-10-11T11:23:29.648 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126003.log 2023-10-11T11:23:29.658 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.126003.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.126003.log.gz 2023-10-11T11:23:29.658 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.128001.log 2023-10-11T11:23:29.681 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.128001.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117808.log 2023-10-11T11:23:29.681 INFO:teuthology.orchestra.run.smithi111.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.128001.log.gz 2023-10-11T11:23:29.681 INFO:teuthology.orchestra.run.smithi111.stderr: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.1.121541.log.gz 2023-10-11T11:23:29.682 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.117808.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.117808.log.gz 2023-10-11T11:23:29.682 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110357.log 2023-10-11T11:23:29.682 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.110357.log: 88.2% -- replaced with /var/log/ceph/cluster1-client.mirror.110357.log.gz 2023-10-11T11:23:29.682 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.146801.log 2023-10-11T11:23:29.682 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.admin.146801.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.146801.log.gz 2023-10-11T11:23:29.683 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125971.log 2023-10-11T11:23:29.683 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.125971.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.125971.log.gz 2023-10-11T11:23:29.683 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.142837.log 2023-10-11T11:23:29.683 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.142837.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.142837.log.gz 2023-10-11T11:23:29.683 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116107.log 2023-10-11T11:23:29.684 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.116107.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.116107.log.gz 2023-10-11T11:23:29.684 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.132515.log 2023-10-11T11:23:29.684 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.132515.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.132515.log.gz 2023-10-11T11:23:29.684 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.127763.log 2023-10-11T11:23:29.684 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.127763.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.127763.log.gz 2023-10-11T11:23:29.684 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.126500.log 2023-10-11T11:23:29.685 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.126500.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.126500.log.gz 2023-10-11T11:23:29.688 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.129808.log 2023-10-11T11:23:29.689 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.129808.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.129808.log.gz 2023-10-11T11:23:29.689 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.143893.log 2023-10-11T11:23:29.689 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.admin.143893.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.143893.log.gz 2023-10-11T11:23:29.690 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.114154.log 2023-10-11T11:23:29.690 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.114154.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.116922.log.gz 2023-10-11T11:23:29.690 INFO:teuthology.orchestra.run.smithi111.stderr: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.114154.log.gz 2023-10-11T11:23:29.690 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.141276.log 2023-10-11T11:23:29.701 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.141276.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.141276.log.gz 2023-10-11T11:23:29.702 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.110333.log 2023-10-11T11:23:29.702 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.110333.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.110333.log.gz 2023-10-11T11:23:29.702 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112991.log 2023-10-11T11:23:29.702 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.112991.log: 89.3% -- replaced with /var/log/ceph/cluster1-client.mirror.112991.log.gz 2023-10-11T11:23:29.703 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117954.log 2023-10-11T11:23:29.703 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.117954.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.117954.log.gz 2023-10-11T11:23:29.703 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.109821.log 2023-10-11T11:23:29.703 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.125630.log 2023-10-11T11:23:29.704 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.109821.log: /var/log/ceph/cluster2-client.mirror.125630.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.109821.log.gz 2023-10-11T11:23:29.704 INFO:teuthology.orchestra.run.smithi111.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.125630.log.gz 2023-10-11T11:23:29.704 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112630.log 2023-10-11T11:23:29.704 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.112630.log: 88.2% -- replaced with /var/log/ceph/cluster1-client.mirror.112630.log.gz 2023-10-11T11:23:29.704 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116067.log 2023-10-11T11:23:29.705 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.116067.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.114572.log 2023-10-11T11:23:29.705 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.114572.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.114572.log.gz 2023-10-11T11:23:29.705 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.105808.log 2023-10-11T11:23:29.707 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.admin.105808.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.105808.log.gz 2023-10-11T11:23:29.707 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.122753.log 2023-10-11T11:23:29.707 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.122753.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.122753.log.gz 2023-10-11T11:23:29.708 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110381.log 2023-10-11T11:23:29.708 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.110381.log: 88.3% -- replaced with /var/log/ceph/cluster1-client.mirror.110381.log.gz 2023-10-11T11:23:29.708 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127335.log 2023-10-11T11:23:29.709 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.127335.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.127335.log.gz 2023-10-11T11:23:29.709 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138617.log 2023-10-11T11:23:29.719 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.138617.log: 2.6% -- replaced with /var/log/ceph/cluster1-client.mirror.138617.log.gz 2023-10-11T11:23:29.719 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.144399.log 2023-10-11T11:23:29.720 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.admin.144399.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.144399.log.gz 2023-10-11T11:23:29.721 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.112694.log 2023-10-11T11:23:29.723 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.112694.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.129716.log 2023-10-11T11:23:29.725 INFO:teuthology.orchestra.run.smithi111.stderr: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.112694.log.gz 2023-10-11T11:23:29.734 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.129716.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.129716.log.gz 2023-10-11T11:23:29.734 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.141232.log 2023-10-11T11:23:29.744 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.141232.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.117996.log.gz 2023-10-11T11:23:29.745 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.120643.log 2023-10-11T11:23:29.746 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.120643.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.120643.log.gz 2023-10-11T11:23:29.746 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123357.log 2023-10-11T11:23:29.746 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.123357.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.123357.log.gz 2023-10-11T11:23:29.747 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.142434.log 2023-10-11T11:23:29.748 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.142434.log: 91.3% -- replaced with /var/log/ceph/cluster2-client.mirror.142434.log.gz 2023-10-11T11:23:29.748 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110538.log 2023-10-11T11:23:29.749 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.110538.log: 88.3% -- replaced with /var/log/ceph/cluster1-client.mirror.110538.log.gz 2023-10-11T11:23:29.749 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.132850.log 2023-10-11T11:23:29.750 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.132850.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.132850.log.gz 2023-10-11T11:23:29.750 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119058.log 2023-10-11T11:23:29.755 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.119058.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.141989.log 2023-10-11T11:23:29.783 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.141989.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.141989.log.gz 2023-10-11T11:23:29.783 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118892.log 2023-10-11T11:23:29.794 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.118892.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.128610.log 2023-10-11T11:23:29.794 INFO:teuthology.orchestra.run.smithi111.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.118892.log.gz 2023-10-11T11:23:29.805 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.128610.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116526.log 2023-10-11T11:23:29.805 INFO:teuthology.orchestra.run.smithi111.stderr: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.128610.log.gz 2023-10-11T11:23:29.816 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.116526.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.116526.log.gz 2023-10-11T11:23:29.817 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.109351.log 2023-10-11T11:23:29.842 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.109351.log: 91.3% -- replaced with /var/log/ceph/cluster2-client.mirror.109351.log.gz 2023-10-11T11:23:29.842 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.131309.log 2023-10-11T11:23:29.853 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.131309.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.131309.log.gz 2023-10-11T11:23:29.853 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124797.log 2023-10-11T11:23:29.864 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.124797.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.124797.log.gz 2023-10-11T11:23:29.864 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.142649.log 2023-10-11T11:23:29.886 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.142649.log: 91.2% -- replaced with /var/log/ceph/cluster2-client.mirror.142649.log.gz 2023-10-11T11:23:29.886 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127932.log 2023-10-11T11:23:29.897 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.127932.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.127932.log.gz 2023-10-11T11:23:29.897 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.110668.log 2023-10-11T11:23:29.908 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.110668.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127835.log 2023-10-11T11:23:29.919 INFO:teuthology.orchestra.run.smithi111.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.110668.log.gz 2023-10-11T11:23:29.930 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.127835.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.127835.log.gz 2023-10-11T11:23:29.930 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.135395.log 2023-10-11T11:23:29.941 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.135395.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.126589.log 2023-10-11T11:23:29.941 INFO:teuthology.orchestra.run.smithi111.stderr: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.135395.log.gz 2023-10-11T11:23:29.950 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.135519.log 2023-10-11T11:23:29.951 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.144307.log 2023-10-11T11:23:29.951 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.115625.log 2023-10-11T11:23:29.952 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.126589.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.126589.log.gz 2023-10-11T11:23:29.952 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.142413.log 2023-10-11T11:23:29.953 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.135788.log 2023-10-11T11:23:29.954 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.107426.log 2023-10-11T11:23:29.955 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111496.log 2023-10-11T11:23:29.956 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.132898.log 2023-10-11T11:23:29.957 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.126157.log 2023-10-11T11:23:29.957 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.135457.log 2023-10-11T11:23:29.958 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136725.log 2023-10-11T11:23:29.959 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.141945.log 2023-10-11T11:23:29.963 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.105577.log 2023-10-11T11:23:29.964 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112043.log 2023-10-11T11:23:29.964 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117256.log 2023-10-11T11:23:29.964 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112606.log 2023-10-11T11:23:29.964 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.145465.log 2023-10-11T11:23:29.965 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.131196.log 2023-10-11T11:23:29.966 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.123241.log 2023-10-11T11:23:29.966 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.141483.log 2023-10-11T11:23:29.968 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.143146.log 2023-10-11T11:23:29.968 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.130018.log 2023-10-11T11:23:29.969 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.105576.log 2023-10-11T11:23:29.969 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.126815.log 2023-10-11T11:23:29.970 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.141577.log 2023-10-11T11:23:29.971 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124454.log 2023-10-11T11:23:29.971 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.131105.log 2023-10-11T11:23:29.972 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118450.log 2023-10-11T11:23:29.973 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137513.log 2023-10-11T11:23:29.973 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.135271.log 2023-10-11T11:23:29.974 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.134300.log 2023-10-11T11:23:29.975 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110134.log 2023-10-11T11:23:29.976 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.134534.log 2023-10-11T11:23:29.977 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.129582.log 2023-10-11T11:23:30.009 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.135519.log: /var/log/ceph/cluster2-client.admin.144307.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.144307.log.gz 2023-10-11T11:23:30.009 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.115625.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.135519.log.gz 2023-10-11T11:23:30.009 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.142413.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.142413.log.gz 2023-10-11T11:23:30.009 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.135788.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.135788.log.gz 2023-10-11T11:23:30.010 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.admin.107426.log: 80.2% -- replaced with /var/log/ceph/cluster2-client.admin.107426.log.gz 2023-10-11T11:23:30.010 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.111496.log: 88.7% -- replaced with /var/log/ceph/cluster1-client.mirror.111496.log.gz 2023-10-11T11:23:30.010 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.132898.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.132898.log.gz 2023-10-11T11:23:30.011 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.126157.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.126157.log.gz 2023-10-11T11:23:30.011 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.135457.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.135457.log.gz 2023-10-11T11:23:30.011 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.136725.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.136725.log.gz 2023-10-11T11:23:30.012 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.141945.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.141945.log.gz 2023-10-11T11:23:30.012 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.admin.105577.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.105577.log.gz 2023-10-11T11:23:30.012 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.112043.log: /var/log/ceph/cluster1-client.mirror.129582.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.129582.log.gz 2023-10-11T11:23:30.012 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.114920.log 2023-10-11T11:23:30.013 INFO:teuthology.orchestra.run.smithi111.stderr: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.112043.log.gz 2023-10-11T11:23:30.013 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.117256.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.117256.log.gz 2023-10-11T11:23:30.014 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.112606.log: 89.1% -- replaced with /var/log/ceph/cluster1-client.mirror.112606.log.gz 2023-10-11T11:23:30.014 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.admin.145465.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.145465.log.gz 2023-10-11T11:23:30.014 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.131196.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.131196.log.gz 2023-10-11T11:23:30.015 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.123241.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.123241.log.gz 2023-10-11T11:23:30.015 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.141483.log: /var/log/ceph/cluster2-client.mirror.114920.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.114920.log.gz 2023-10-11T11:23:30.015 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.134341.log 2023-10-11T11:23:30.016 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.admin.143146.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.143146.log.gz 2023-10-11T11:23:30.016 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.130018.log: 86.4% -- replaced with /var/log/ceph/cluster1-client.mirror.130018.log.gz 2023-10-11T11:23:30.016 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.admin.105576.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.105576.log.gz 2023-10-11T11:23:30.016 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.126815.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.126815.log.gz 2023-10-11T11:23:30.017 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.141577.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.141577.log.gz 2023-10-11T11:23:30.017 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.124454.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.124454.log.gz 2023-10-11T11:23:30.018 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.131105.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.131105.log.gz 2023-10-11T11:23:30.018 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.118450.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.118450.log.gz 2023-10-11T11:23:30.018 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.137513.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.137513.log.gz 2023-10-11T11:23:30.019 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.135271.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.135271.log.gz 2023-10-11T11:23:30.019 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.134300.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.134300.log.gz 2023-10-11T11:23:30.020 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.110134.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.115980.log.gz 2023-10-11T11:23:30.025 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.134341.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.134341.log.gz 2023-10-11T11:23:30.025 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119861.log 2023-10-11T11:23:30.026 INFO:teuthology.orchestra.run.smithi111.stderr: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.110134.log.gz 2023-10-11T11:23:30.026 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.134534.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.134534.log.gz 2023-10-11T11:23:30.036 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.119861.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.119861.log.gz 2023-10-11T11:23:30.037 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.114980.log 2023-10-11T11:23:30.037 INFO:teuthology.orchestra.run.smithi111.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.119058.log.gz 2023-10-11T11:23:30.047 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.114980.log: 80.3% -- replaced with /var/log/ceph/cluster2-client.mirror.114980.log.gz 2023-10-11T11:23:30.048 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.120570.log 2023-10-11T11:23:30.048 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.120570.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.120570.log.gz 2023-10-11T11:23:30.049 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.114343.log 2023-10-11T11:23:30.050 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.114343.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.114343.log.gz 2023-10-11T11:23:30.050 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137355.log 2023-10-11T11:23:30.051 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.137355.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.137355.log.gz 2023-10-11T11:23:30.051 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.125045.log 2023-10-11T11:23:30.051 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.125045.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.125045.log.gz 2023-10-11T11:23:30.052 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137265.log 2023-10-11T11:23:30.054 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.137265.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.137265.log.gz 2023-10-11T11:23:30.054 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.107127.log 2023-10-11T11:23:30.055 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.admin.107127.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.107127.log.gz 2023-10-11T11:23:30.055 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.141173.log 2023-10-11T11:23:30.061 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.141173.log: 80.5% -- replaced with /var/log/ceph/cluster2-client.mirror.141173.log.gz 2023-10-11T11:23:30.061 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.134509.log 2023-10-11T11:23:30.061 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.134509.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.134509.log.gz 2023-10-11T11:23:30.062 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.113931.log 2023-10-11T11:23:30.062 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.113931.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.116067.log.gz 2023-10-11T11:23:30.062 INFO:teuthology.orchestra.run.smithi111.stderr: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.113931.log.gz 2023-10-11T11:23:30.063 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.140804.log 2023-10-11T11:23:30.063 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.140804.log: 80.5% -- replaced with /var/log/ceph/cluster2-client.mirror.140804.log.gz 2023-10-11T11:23:30.063 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119456.log 2023-10-11T11:23:30.063 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.119456.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.119456.log.gz 2023-10-11T11:23:30.063 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.123146.log 2023-10-11T11:23:30.064 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.123146.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.123146.log.gz 2023-10-11T11:23:30.064 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.134699.log 2023-10-11T11:23:30.070 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.134699.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.134699.log.gz 2023-10-11T11:23:30.070 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.136096.log 2023-10-11T11:23:30.070 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.136096.log: 86.2% -- replaced with /var/log/ceph/cluster2-client.mirror.136096.log.gz 2023-10-11T11:23:30.070 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2.audit.log 2023-10-11T11:23:30.071 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2.audit.log: 92.5% -- replaced with /var/log/ceph/cluster2.audit.log.gz 2023-10-11T11:23:30.071 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.131378.log 2023-10-11T11:23:30.071 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.131378.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.131378.log.gz 2023-10-11T11:23:30.071 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.133698.log 2023-10-11T11:23:30.072 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.133698.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.133698.log.gz 2023-10-11T11:23:30.072 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118973.log 2023-10-11T11:23:30.073 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.118973.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.118973.log.gz 2023-10-11T11:23:30.074 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.145787.log 2023-10-11T11:23:30.074 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.admin.145787.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.145787.log.gz 2023-10-11T11:23:30.074 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119548.log 2023-10-11T11:23:30.074 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.119548.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.119548.log.gz 2023-10-11T11:23:30.074 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.129760.log 2023-10-11T11:23:30.075 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.129760.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.129760.log.gz 2023-10-11T11:23:30.075 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.128319.log 2023-10-11T11:23:30.085 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.128319.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.128319.log.gz 2023-10-11T11:23:30.086 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119248.log 2023-10-11T11:23:30.086 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.119248.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.119248.log.gz 2023-10-11T11:23:30.086 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.126090.log 2023-10-11T11:23:30.086 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.126090.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.126090.log.gz 2023-10-11T11:23:30.086 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.120004.log 2023-10-11T11:23:30.087 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.120004.log: 80.3% -- replaced with /var/log/ceph/cluster2-client.mirror.120004.log.gz 2023-10-11T11:23:30.087 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.117043.log 2023-10-11T11:23:30.096 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.0.117043.log: 93.3% -- replaced with /var/log/ceph/cluster2-client.mirror.0.117043.log.gz 2023-10-11T11:23:30.096 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116415.log 2023-10-11T11:23:30.096 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.116415.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.116415.log.gz 2023-10-11T11:23:30.096 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122678.log 2023-10-11T11:23:30.097 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.122678.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.122678.log.gz 2023-10-11T11:23:30.097 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121435.log 2023-10-11T11:23:30.097 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.121435.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.121435.log.gz 2023-10-11T11:23:30.097 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.143801.log 2023-10-11T11:23:30.097 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.admin.143801.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.143801.log.gz 2023-10-11T11:23:30.097 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.144261.log 2023-10-11T11:23:30.098 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.admin.144261.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.144261.log.gz 2023-10-11T11:23:30.098 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120483.log 2023-10-11T11:23:30.098 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.120483.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.120483.log.gz 2023-10-11T11:23:30.098 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.131494.log 2023-10-11T11:23:30.098 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.131494.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.131494.log.gz 2023-10-11T11:23:30.099 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125519.log 2023-10-11T11:23:30.099 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.125519.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.125519.log.gz 2023-10-11T11:23:30.099 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.141090.log 2023-10-11T11:23:30.099 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.141090.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.141090.log.gz 2023-10-11T11:23:30.099 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111329.log 2023-10-11T11:23:30.100 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.111329.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.111329.log.gz 2023-10-11T11:23:30.101 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.125201.log 2023-10-11T11:23:30.101 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.125201.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.125201.log.gz 2023-10-11T11:23:30.101 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.140452.log 2023-10-11T11:23:30.101 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.140452.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.140452.log.gz 2023-10-11T11:23:30.101 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117850.log 2023-10-11T11:23:30.102 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.117850.log: 80.2% -- replaced with /var/log/ceph/cluster2-client.mirror.117850.log.gz 2023-10-11T11:23:30.102 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.130422.log 2023-10-11T11:23:30.102 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.130422.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.130422.log.gz 2023-10-11T11:23:30.102 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.130129.log 2023-10-11T11:23:30.112 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.130129.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.130129.log.gz 2023-10-11T11:23:30.112 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114217.log 2023-10-11T11:23:30.112 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.114217.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.144077.log 2023-10-11T11:23:30.112 INFO:teuthology.orchestra.run.smithi111.stderr: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.114217.log.gz 2023-10-11T11:23:30.113 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.admin.144077.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.144077.log.gz 2023-10-11T11:23:30.113 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138233.log 2023-10-11T11:23:30.113 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.138233.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.138233.log.gz 2023-10-11T11:23:30.113 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126705.log 2023-10-11T11:23:30.113 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.126705.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.126705.log.gz 2023-10-11T11:23:30.114 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.127537.log 2023-10-11T11:23:30.123 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.127537.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.127537.log.gz 2023-10-11T11:23:30.123 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.111355.log 2023-10-11T11:23:30.123 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.111355.log: 91.6% -- replaced with /var/log/ceph/cluster2-client.mirror.111355.log.gz 2023-10-11T11:23:30.123 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.112758.log 2023-10-11T11:23:30.124 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.112758.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.134406.log 2023-10-11T11:23:30.133 INFO:teuthology.orchestra.run.smithi111.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.141232.log.gz 2023-10-11T11:23:30.134 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.134406.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.134406.log.gz 2023-10-11T11:23:30.134 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113736.log 2023-10-11T11:23:30.134 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.113736.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121500.log 2023-10-11T11:23:30.134 INFO:teuthology.orchestra.run.smithi111.stderr: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.113736.log.gz 2023-10-11T11:23:30.136 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.121500.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.121500.log.gz 2023-10-11T11:23:30.136 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.110873.log 2023-10-11T11:23:30.136 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.110873.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.110873.log.gz 2023-10-11T11:23:30.137 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.115017.log 2023-10-11T11:23:30.137 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.115017.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.115017.log.gz 2023-10-11T11:23:30.138 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109466.log 2023-10-11T11:23:30.138 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.109466.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.109466.log.gz 2023-10-11T11:23:30.138 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122637.log 2023-10-11T11:23:30.140 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.122637.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.122637.log.gz 2023-10-11T11:23:30.140 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.141253.log 2023-10-11T11:23:30.141 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.141253.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.141253.log.gz 2023-10-11T11:23:30.141 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.128138.log 2023-10-11T11:23:30.143 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.128138.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.128138.log.gz 2023-10-11T11:23:30.143 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108442.log 2023-10-11T11:23:30.144 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.108442.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.108442.log.gz 2023-10-11T11:23:30.145 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.108515.log 2023-10-11T11:23:30.146 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.108515.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.108515.log.gz 2023-10-11T11:23:30.146 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.133111.log 2023-10-11T11:23:30.147 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.133111.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.133111.log.gz 2023-10-11T11:23:30.147 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137378.log 2023-10-11T11:23:30.148 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.137378.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.137378.log.gz 2023-10-11T11:23:30.148 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119607.log 2023-10-11T11:23:30.150 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.119607.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119756.log 2023-10-11T11:23:30.150 INFO:teuthology.orchestra.run.smithi111.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.119607.log.gz 2023-10-11T11:23:30.151 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.119756.log: 80.3% -- replaced with /var/log/ceph/cluster2-client.mirror.119756.log.gz 2023-10-11T11:23:30.151 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.140394.log 2023-10-11T11:23:30.154 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.140394.log: 91.1% -- replaced with /var/log/ceph/cluster2-client.mirror.140394.log.gz 2023-10-11T11:23:30.154 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.125677.log 2023-10-11T11:23:30.155 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.125677.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.125677.log.gz 2023-10-11T11:23:30.156 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.140611.log 2023-10-11T11:23:30.157 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.140611.log: 81.9% -- replaced with /var/log/ceph/cluster1-client.mirror.140611.log.gz 2023-10-11T11:23:30.157 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109028.log 2023-10-11T11:23:30.158 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.109028.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.109028.log.gz 2023-10-11T11:23:30.158 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.113974.log 2023-10-11T11:23:30.161 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.113974.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137175.log 2023-10-11T11:23:30.164 INFO:teuthology.orchestra.run.smithi111.stderr: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.113974.log.gz 2023-10-11T11:23:30.165 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.137175.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.137175.log.gz 2023-10-11T11:23:30.165 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.143755.log 2023-10-11T11:23:30.166 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.admin.143755.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.143755.log.gz 2023-10-11T11:23:30.166 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.132265.log 2023-10-11T11:23:30.167 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.132265.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.132265.log.gz 2023-10-11T11:23:30.167 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.146065.log 2023-10-11T11:23:30.168 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.admin.146065.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.146065.log.gz 2023-10-11T11:23:30.168 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.112510.log 2023-10-11T11:23:30.169 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.112510.log: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.112510.log.gz 2023-10-11T11:23:30.169 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.108412.log 2023-10-11T11:23:30.170 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.admin.108412.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.108412.log.gz 2023-10-11T11:23:30.170 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136680.log 2023-10-11T11:23:30.171 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.136680.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.136680.log.gz 2023-10-11T11:23:30.171 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119013.log 2023-10-11T11:23:30.172 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.119013.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.119013.log.gz 2023-10-11T11:23:30.172 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.128271.log 2023-10-11T11:23:30.173 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.128271.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.128271.log.gz 2023-10-11T11:23:30.174 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110969.log 2023-10-11T11:23:30.175 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.110969.log: 88.5% -- replaced with /var/log/ceph/cluster1-client.mirror.110969.log.gz 2023-10-11T11:23:30.175 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.142860.log 2023-10-11T11:23:30.175 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.142860.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.142860.log.gz 2023-10-11T11:23:30.176 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.111121.log 2023-10-11T11:23:30.180 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.111121.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138525.log 2023-10-11T11:23:30.201 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.138525.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.138525.log.gz 2023-10-11T11:23:30.201 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119815.log 2023-10-11T11:23:30.214 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.119815.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.130792.log 2023-10-11T11:23:30.227 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.130792.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.130792.log.gz 2023-10-11T11:23:30.227 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.141600.log 2023-10-11T11:23:30.240 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.141600.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.141600.log.gz 2023-10-11T11:23:30.240 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122149.log 2023-10-11T11:23:30.253 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.122149.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.122149.log.gz 2023-10-11T11:23:30.253 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.133090.log 2023-10-11T11:23:30.278 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.133090.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.133090.log.gz 2023-10-11T11:23:30.278 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.141968.log 2023-10-11T11:23:30.291 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.141968.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.141968.log.gz 2023-10-11T11:23:30.291 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138165.log 2023-10-11T11:23:30.302 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.138165.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.138165.log.gz 2023-10-11T11:23:30.302 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.126609.log 2023-10-11T11:23:30.315 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.126609.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.126609.log.gz 2023-10-11T11:23:30.315 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125834.log 2023-10-11T11:23:30.326 INFO:teuthology.orchestra.run.smithi111.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.142361.log.gz 2023-10-11T11:23:30.341 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118813.log 2023-10-11T11:23:30.352 INFO:teuthology.orchestra.run.smithi111.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.112758.log.gz 2023-10-11T11:23:30.352 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.118813.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.118813.log.gz 2023-10-11T11:23:30.353 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.143196.log 2023-10-11T11:23:30.353 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.admin.143196.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.143196.log.gz 2023-10-11T11:23:30.353 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110921.log 2023-10-11T11:23:30.354 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.110921.log: 88.4% -- replaced with /var/log/ceph/cluster1-client.mirror.110921.log.gz 2023-10-11T11:23:30.358 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.126363.log 2023-10-11T11:23:30.358 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.126363.log: /var/log/ceph/cluster1-client.mirror.125834.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.126363.log.gz 2023-10-11T11:23:30.359 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118016.log 2023-10-11T11:23:30.359 INFO:teuthology.orchestra.run.smithi111.stderr: 92.1% -- replaced with /var/log/ceph/cluster1-client.mirror.125834.log.gz 2023-10-11T11:23:30.359 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.118016.log: 80.3% -- replaced with /var/log/ceph/cluster2-client.mirror.118016.log.gz 2023-10-11T11:23:30.359 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.110626.log 2023-10-11T11:23:30.363 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.110626.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.135664.log 2023-10-11T11:23:30.364 INFO:teuthology.orchestra.run.smithi111.stderr: 93.3% -- replaced with /var/log/ceph/cluster2-client.mirror.110626.log.gz 2023-10-11T11:23:30.365 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.135664.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.135664.log.gz 2023-10-11T11:23:30.368 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.108468.log 2023-10-11T11:23:30.377 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.108468.log: 90.5% -- replaced with /var/log/ceph/cluster2-client.mirror.108468.log.gz 2023-10-11T11:23:30.377 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.128904.log 2023-10-11T11:23:30.378 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.128904.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.128904.log.gz 2023-10-11T11:23:30.379 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.111882.log 2023-10-11T11:23:30.379 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.111882.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.111882.log.gz 2023-10-11T11:23:30.379 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.145927.log 2023-10-11T11:23:30.379 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.admin.145927.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.145927.log.gz 2023-10-11T11:23:30.379 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.105153.log 2023-10-11T11:23:30.380 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.admin.105153.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.105153.log.gz 2023-10-11T11:23:30.380 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.114997.log 2023-10-11T11:23:30.380 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.114997.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.114997.log.gz 2023-10-11T11:23:30.380 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119327.log 2023-10-11T11:23:30.380 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.119327.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.119327.log.gz 2023-10-11T11:23:30.381 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.141297.log 2023-10-11T11:23:30.381 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.141297.log: 91.4% -- replaced with /var/log/ceph/cluster2-client.mirror.141297.log.gz 2023-10-11T11:23:30.381 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121776.log 2023-10-11T11:23:30.382 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.121776.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.121776.log.gz 2023-10-11T11:23:30.382 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137288.log 2023-10-11T11:23:30.382 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.137288.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.137288.log.gz 2023-10-11T11:23:30.382 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.108772.log 2023-10-11T11:23:30.383 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.admin.108772.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.108772.log.gz 2023-10-11T11:23:30.383 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.146157.log 2023-10-11T11:23:30.383 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.admin.146157.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.146157.log.gz 2023-10-11T11:23:30.384 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.131289.log 2023-10-11T11:23:30.385 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.131289.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.131289.log.gz 2023-10-11T11:23:30.385 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.143050.log 2023-10-11T11:23:30.385 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.143050.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.143050.log.gz 2023-10-11T11:23:30.386 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.129606.log 2023-10-11T11:23:30.387 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.129606.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.129606.log.gz 2023-10-11T11:23:30.387 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114383.log 2023-10-11T11:23:30.398 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.114383.log: 95.4% -- replaced with /var/log/ceph/cluster1-client.mirror.114383.log.gz 2023-10-11T11:23:30.398 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.126274.log 2023-10-11T11:23:30.400 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.126274.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.142629.log 2023-10-11T11:23:30.400 INFO:teuthology.orchestra.run.smithi111.stderr: 92.0% -- replaced with /var/log/ceph/cluster2-client.mirror.126274.log.gz 2023-10-11T11:23:30.401 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.142629.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.142629.log.gz 2023-10-11T11:23:30.402 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.133781.log 2023-10-11T11:23:30.403 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.133781.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.133781.log.gz 2023-10-11T11:23:30.403 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.111014.log 2023-10-11T11:23:30.409 INFO:teuthology.orchestra.run.smithi111.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.141483.log.gz 2023-10-11T11:23:30.409 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.111014.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.109962.log 2023-10-11T11:23:30.410 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.109962.log: 93.3% -- replaced with /var/log/ceph/cluster2-client.mirror.111014.log.gz 2023-10-11T11:23:30.410 INFO:teuthology.orchestra.run.smithi111.stderr: 93.3% -- replaced with /var/log/ceph/cluster2-client.mirror.109962.log.gz 2023-10-11T11:23:30.410 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116981.log 2023-10-11T11:23:30.411 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.116981.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.116981.log.gz 2023-10-11T11:23:30.411 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138570.log 2023-10-11T11:23:30.412 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.138570.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.138570.log.gz 2023-10-11T11:23:30.412 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.117571.log 2023-10-11T11:23:30.415 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.0.117571.log: 92.7% -- replaced with /var/log/ceph/cluster1-client.mirror.0.117571.log.gz 2023-10-11T11:23:30.416 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.131902.log 2023-10-11T11:23:30.418 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.131902.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.131902.log.gz 2023-10-11T11:23:30.418 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.143171.log 2023-10-11T11:23:30.419 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.admin.143171.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.143171.log.gz 2023-10-11T11:23:30.419 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117236.log 2023-10-11T11:23:30.420 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.117236.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.117236.log.gz 2023-10-11T11:23:30.420 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.132063.log 2023-10-11T11:23:30.421 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.132063.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.132063.log.gz 2023-10-11T11:23:30.421 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.130652.log 2023-10-11T11:23:30.422 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.130652.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.130652.log.gz 2023-10-11T11:23:30.423 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.129950.log 2023-10-11T11:23:30.424 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.129950.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.129950.log.gz 2023-10-11T11:23:30.424 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137198.log 2023-10-11T11:23:30.425 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.137198.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.137198.log.gz 2023-10-11T11:23:30.425 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.140763.log 2023-10-11T11:23:30.426 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.140763.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.140763.log.gz 2023-10-11T11:23:30.426 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.115645.log 2023-10-11T11:23:30.427 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.115645.log: 80.4% -- replaced with /var/log/ceph/cluster2-client.mirror.115645.log.gz 2023-10-11T11:23:30.427 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.132491.log 2023-10-11T11:23:30.433 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.132491.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.132491.log.gz 2023-10-11T11:23:30.434 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.131446.log 2023-10-11T11:23:30.434 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.131446.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.131446.log.gz 2023-10-11T11:23:30.434 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118255.log 2023-10-11T11:23:30.435 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.118255.log: 80.3% -- replaced with /var/log/ceph/cluster2-client.mirror.118255.log.gz 2023-10-11T11:23:30.435 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.120314.log 2023-10-11T11:23:30.435 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.120314.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.120314.log.gz 2023-10-11T11:23:30.436 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117306.log 2023-10-11T11:23:30.436 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.117306.log: 80.4% -- replaced with /var/log/ceph/cluster2-client.mirror.117306.log.gz 2023-10-11T11:23:30.436 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116734.log 2023-10-11T11:23:30.443 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.115167.log 2023-10-11T11:23:30.443 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.116734.log: /var/log/ceph/cluster2-client.mirror.115167.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.115167.log.gz 2023-10-11T11:23:30.444 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.141190.log 2023-10-11T11:23:30.444 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.141190.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.141190.log.gz 2023-10-11T11:23:30.445 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.122890.log 2023-10-11T11:23:30.445 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.122890.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.126048.log 2023-10-11T11:23:30.445 INFO:teuthology.orchestra.run.smithi111.stderr: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.122890.log.gz 2023-10-11T11:23:30.446 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.126048.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.126048.log.gz 2023-10-11T11:23:30.446 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112002.log 2023-10-11T11:23:30.447 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.112002.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.112002.log.gz 2023-10-11T11:23:30.447 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.141152.log 2023-10-11T11:23:30.454 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.141152.log: 91.2% -- replaced with /var/log/ceph/cluster2-client.mirror.141152.log.gz 2023-10-11T11:23:30.454 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125293.log 2023-10-11T11:23:30.460 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.125293.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.125293.log.gz 2023-10-11T11:23:30.460 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119590.log 2023-10-11T11:23:30.461 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.119590.log: 80.3% -- replaced with /var/log/ceph/cluster2-client.mirror.119590.log.gz 2023-10-11T11:23:30.462 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118314.log 2023-10-11T11:23:30.472 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.118314.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.115625.log.gz 2023-10-11T11:23:30.472 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.130696.log 2023-10-11T11:23:30.473 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.130696.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.130696.log.gz 2023-10-11T11:23:30.473 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138300.log 2023-10-11T11:23:30.475 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.138300.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.138300.log.gz 2023-10-11T11:23:30.475 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.110690.log 2023-10-11T11:23:30.490 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.110690.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118211.log 2023-10-11T11:23:30.502 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.118211.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.118211.log.gz 2023-10-11T11:23:30.502 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.128771.log 2023-10-11T11:23:30.514 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.128771.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.128771.log.gz 2023-10-11T11:23:30.515 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124593.log 2023-10-11T11:23:30.521 INFO:teuthology.orchestra.run.smithi111.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.119815.log.gz 2023-10-11T11:23:30.523 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.124593.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.124593.log.gz 2023-10-11T11:23:30.523 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.115466.log 2023-10-11T11:23:30.523 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.115466.log: 80.3% -- replaced with /var/log/ceph/cluster2-client.mirror.115466.log.gz 2023-10-11T11:23:30.524 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118551.log 2023-10-11T11:23:30.525 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.118551.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.118551.log.gz 2023-10-11T11:23:30.525 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.108726.log 2023-10-11T11:23:30.526 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.108726.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.108726.log.gz 2023-10-11T11:23:30.526 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123166.log 2023-10-11T11:23:30.529 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.123166.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.123166.log.gz 2023-10-11T11:23:30.529 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136748.log 2023-10-11T11:23:30.529 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.136748.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.136748.log.gz 2023-10-11T11:23:30.530 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113284.log 2023-10-11T11:23:30.531 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.113284.log: 88.3% -- replaced with /var/log/ceph/cluster1-client.mirror.113284.log.gz 2023-10-11T11:23:30.531 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.134072.log 2023-10-11T11:23:30.533 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.134072.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.134072.log.gz 2023-10-11T11:23:30.534 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.123610.log 2023-10-11T11:23:30.536 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.123610.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.123610.log.gz 2023-10-11T11:23:30.536 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109869.log 2023-10-11T11:23:30.537 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.109869.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.109869.log.gz 2023-10-11T11:23:30.537 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.134575.log 2023-10-11T11:23:30.540 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.134575.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.135374.log 2023-10-11T11:23:30.541 INFO:teuthology.orchestra.run.smithi111.stderr: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.134575.log.gz 2023-10-11T11:23:30.544 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.135374.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.135374.log.gz 2023-10-11T11:23:30.545 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138278.log 2023-10-11T11:23:30.545 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster1-client.mirror.138278.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.138278.log.gz 2023-10-11T11:23:30.545 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118717.log 2023-10-11T11:23:30.546 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.118717.log: 80.3% -- replaced with /var/log/ceph/cluster2-client.mirror.118717.log.gz 2023-10-11T11:23:30.546 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.134468.log 2023-10-11T11:23:30.547 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.134468.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.134468.log.gz 2023-10-11T11:23:30.548 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.1.128815.log 2023-10-11T11:23:30.550 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.1.128815.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.1.128815.log.gz 2023-10-11T11:23:30.551 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.125923.log 2023-10-11T11:23:30.552 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.125923.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.125923.log.gz 2023-10-11T11:23:30.552 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.132217.log 2023-10-11T11:23:30.553 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.132217.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.132217.log.gz 2023-10-11T11:23:30.553 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117444.log 2023-10-11T11:23:30.633 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cluster2-client.mirror.117444.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.118314.log.gz 2023-10-11T11:23:30.664 INFO:teuthology.orchestra.run.smithi111.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.111121.log.gz 2023-10-11T11:23:30.732 INFO:teuthology.orchestra.run.smithi111.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.117444.log.gz 2023-10-11T11:23:30.835 INFO:teuthology.orchestra.run.smithi111.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.116734.log.gz 2023-10-11T11:23:30.918 INFO:teuthology.orchestra.run.smithi111.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.110690.log.gz 2023-10-11T11:23:33.636 INFO:teuthology.orchestra.run.smithi050.stderr: 94.4% -- replaced with /var/log/ceph/cluster1-osd.1.log.gz 2023-10-11T11:23:34.359 INFO:teuthology.orchestra.run.smithi050.stderr: 94.5% -- replaced with /var/log/ceph/cluster1-osd.2.log.gz 2023-10-11T11:23:34.740 INFO:teuthology.orchestra.run.smithi050.stderr: 94.4% -- replaced with /var/log/ceph/cluster1-osd.0.log.gz 2023-10-11T11:23:34.742 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-11T11:23:34.743 INFO:teuthology.orchestra.run.smithi050.stderr:real 0m9.503s 2023-10-11T11:23:34.743 INFO:teuthology.orchestra.run.smithi050.stderr:user 0m27.531s 2023-10-11T11:23:34.743 INFO:teuthology.orchestra.run.smithi050.stderr:sys 0m1.030s 2023-10-11T11:24:11.377 INFO:teuthology.orchestra.run.smithi111.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-osd.1.log.gz 2023-10-11T11:24:12.232 INFO:teuthology.orchestra.run.smithi111.stderr: 93.6% -- replaced with /var/log/ceph/cluster2-osd.2.log.gz 2023-10-11T11:24:18.784 INFO:teuthology.orchestra.run.smithi111.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-osd.0.log.gz 2023-10-11T11:24:18.786 INFO:teuthology.orchestra.run.smithi111.stderr: 2023-10-11T11:24:18.786 INFO:teuthology.orchestra.run.smithi111.stderr:real 0m53.522s 2023-10-11T11:24:18.786 INFO:teuthology.orchestra.run.smithi111.stderr:user 2m44.908s 2023-10-11T11:24:18.787 INFO:teuthology.orchestra.run.smithi111.stderr:sys 0m6.965s 2023-10-11T11:24:18.787 INFO:tasks.ceph:Archiving logs... 2023-10-11T11:24:18.788 DEBUG:teuthology.misc:Transferring archived files from smithi050:/var/log/ceph to /home/teuthworker/archive/dis-2023-10-11_10:22:06-rbd-wip-dis-testing-distro-default-smithi/7421102/remote/smithi050/log 2023-10-11T11:24:18.789 DEBUG:teuthology.orchestra.run.smithi050:> sudo tar c -f - -C /var/log/ceph -- . 2023-10-11T11:24:22.859 DEBUG:teuthology.misc:Transferring archived files from smithi111:/var/log/ceph to /home/teuthworker/archive/dis-2023-10-11_10:22:06-rbd-wip-dis-testing-distro-default-smithi/7421102/remote/smithi111/log 2023-10-11T11:24:22.860 DEBUG:teuthology.orchestra.run.smithi111:> sudo tar c -f - -C /var/log/ceph -- . 2023-10-11T11:24:50.705 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-10-11T11:24:50.735 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/task/install/__init__.py", line 626, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_idryomov_ceph_3d099a3859cc211cf8dd2acdaf3194ba0e96ea9c/qa/tasks/ceph.py", line 1950, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_idryomov_ceph_3d099a3859cc211cf8dd2acdaf3194ba0e96ea9c/qa/tasks/ceph_manager.py", line 2743, in wait_for_clean while not self.is_clean(): File "/home/teuthworker/src/github.com_idryomov_ceph_3d099a3859cc211cf8dd2acdaf3194ba0e96ea9c/qa/tasks/ceph_manager.py", line 2679, in is_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_idryomov_ceph_3d099a3859cc211cf8dd2acdaf3194ba0e96ea9c/qa/tasks/ceph_manager.py", line 2340, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_idryomov_ceph_3d099a3859cc211cf8dd2acdaf3194ba0e96ea9c/qa/tasks/ceph_manager.py", line 1595, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_idryomov_ceph_3d099a3859cc211cf8dd2acdaf3194ba0e96ea9c/qa/tasks/ceph_manager.py", line 1586, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi111 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json' 2023-10-11T11:24:50.736 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-10-11T11:24:50.736 DEBUG:teuthology.orchestra.run.smithi050:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-10-11T11:24:50.745 DEBUG:teuthology.orchestra.run.smithi111:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-10-11T11:24:50.820 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-10-11T11:24:50.836 INFO:teuthology.task.clock:Checking final clock skew... 2023-10-11T11:24:50.836 DEBUG:teuthology.orchestra.run.smithi050:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-11T11:24:50.858 DEBUG:teuthology.orchestra.run.smithi111:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-11T11:24:50.875 INFO:teuthology.orchestra.run.smithi050.stderr:bash: ntpq: command not found 2023-10-11T11:24:50.877 INFO:teuthology.orchestra.run.smithi111.stderr:bash: ntpq: command not found 2023-10-11T11:24:50.880 INFO:teuthology.orchestra.run.smithi050.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-10-11T11:24:50.880 INFO:teuthology.orchestra.run.smithi050.stdout:=============================================================================== 2023-10-11T11:24:50.880 INFO:teuthology.orchestra.run.smithi050.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 58 -1961us[-1961us] +/- 91ms 2023-10-11T11:24:50.880 INFO:teuthology.orchestra.run.smithi050.stdout:^* hv02.front.sepia.ceph.com 3 6 377 59 +389us[ +383us] +/- 75ms 2023-10-11T11:24:50.881 INFO:teuthology.orchestra.run.smithi050.stdout:^+ hv03.front.sepia.ceph.com 3 7 377 126 +1456us[+1439us] +/- 80ms 2023-10-11T11:24:50.881 INFO:teuthology.orchestra.run.smithi050.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2023-10-11T11:24:50.883 INFO:teuthology.orchestra.run.smithi111.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-10-11T11:24:50.883 INFO:teuthology.orchestra.run.smithi111.stdout:=============================================================================== 2023-10-11T11:24:50.884 INFO:teuthology.orchestra.run.smithi111.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 62 -1966us[-1966us] +/- 91ms 2023-10-11T11:24:50.884 INFO:teuthology.orchestra.run.smithi111.stdout:^* hv02.front.sepia.ceph.com 3 6 377 62 +380us[ +368us] +/- 75ms 2023-10-11T11:24:50.884 INFO:teuthology.orchestra.run.smithi111.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 60 +1398us[+1398us] +/- 81ms 2023-10-11T11:24:50.884 INFO:teuthology.orchestra.run.smithi111.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2023-10-11T11:24:50.889 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-10-11T11:24:50.904 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-10-11T11:24:50.904 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-10-11T11:24:50.927 DEBUG:teuthology.orchestra.run.smithi050:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-10-11T11:24:51.030 DEBUG:teuthology.orchestra.run.smithi111:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-10-11T11:24:51.123 DEBUG:teuthology.orchestra.run.smithi050:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-11T11:24:51.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-11T11:24:51.160 DEBUG:teuthology.orchestra.run.smithi111:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-11T11:24:51.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-11T11:24:51.192 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-10-11T11:24:51.206 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-10-11T11:24:51.252 INFO:teuthology.task.internal:Duration was 3379.890926 seconds 2023-10-11T11:24:51.258 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-10-11T11:24:51.276 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-10-11T11:24:51.276 DEBUG:teuthology.orchestra.run.smithi050:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-10-11T11:24:51.282 DEBUG:teuthology.orchestra.run.smithi111:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-10-11T11:24:51.328 INFO:teuthology.orchestra.run.smithi111.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-11T11:24:51.329 INFO:teuthology.orchestra.run.smithi050.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-11T11:24:51.603 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-10-11T11:24:51.604 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi050.front.sepia.ceph.com 2023-10-11T11:24:51.604 DEBUG:teuthology.orchestra.run.smithi050:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-10-11T11:24:51.629 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi111.front.sepia.ceph.com 2023-10-11T11:24:51.629 DEBUG:teuthology.orchestra.run.smithi111:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-10-11T11:24:51.664 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-10-11T11:24:51.665 DEBUG:teuthology.orchestra.run.smithi050:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-10-11T11:24:51.672 DEBUG:teuthology.orchestra.run.smithi111:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-10-11T11:24:51.747 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-10-11T11:24:51.748 DEBUG:teuthology.orchestra.run.smithi050:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-10-11T11:24:51.755 DEBUG:teuthology.orchestra.run.smithi111:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-10-11T11:24:51.865 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-10-11T11:24:51.887 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-10-11T11:24:51.888 DEBUG:teuthology.orchestra.run.smithi050:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-10-11T11:24:51.925 DEBUG:teuthology.orchestra.run.smithi111:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-10-11T11:24:51.967 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-10-11T11:24:51.983 DEBUG:teuthology.orchestra.run.smithi050:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-10-11T11:24:51.988 DEBUG:teuthology.orchestra.run.smithi111:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-10-11T11:24:52.013 INFO:teuthology.orchestra.run.smithi050.stdout:kernel.core_pattern = core 2023-10-11T11:24:52.037 INFO:teuthology.orchestra.run.smithi111.stdout:kernel.core_pattern = core 2023-10-11T11:24:52.051 DEBUG:teuthology.orchestra.run.smithi050:> test -e /home/ubuntu/cephtest/archive/coredump 2023-10-11T11:24:52.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-11T11:24:52.096 DEBUG:teuthology.orchestra.run.smithi111:> test -e /home/ubuntu/cephtest/archive/coredump 2023-10-11T11:24:52.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-11T11:24:52.113 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-10-11T11:24:52.128 INFO:teuthology.task.internal:Transferring archived files... 2023-10-11T11:24:52.130 DEBUG:teuthology.misc:Transferring archived files from smithi050:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/dis-2023-10-11_10:22:06-rbd-wip-dis-testing-distro-default-smithi/7421102/remote/smithi050 2023-10-11T11:24:52.130 DEBUG:teuthology.orchestra.run.smithi050:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-10-11T11:24:52.178 DEBUG:teuthology.misc:Transferring archived files from smithi111:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/dis-2023-10-11_10:22:06-rbd-wip-dis-testing-distro-default-smithi/7421102/remote/smithi111 2023-10-11T11:24:52.179 DEBUG:teuthology.orchestra.run.smithi111:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-10-11T11:24:52.234 INFO:teuthology.task.internal:Removing archive directory... 2023-10-11T11:24:52.235 DEBUG:teuthology.orchestra.run.smithi050:> rm -rf -- /home/ubuntu/cephtest/archive 2023-10-11T11:24:52.239 DEBUG:teuthology.orchestra.run.smithi111:> rm -rf -- /home/ubuntu/cephtest/archive 2023-10-11T11:24:52.270 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-10-11T11:24:52.288 INFO:teuthology.task.internal:Not uploading archives. 2023-10-11T11:24:52.288 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-10-11T11:24:52.305 INFO:teuthology.task.internal:Tidying up after the test... 2023-10-11T11:24:52.305 DEBUG:teuthology.orchestra.run.smithi050:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-10-11T11:24:52.315 DEBUG:teuthology.orchestra.run.smithi111:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-10-11T11:24:52.328 INFO:teuthology.orchestra.run.smithi050.stdout: 262307 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Oct 11 11:24 /home/ubuntu/cephtest 2023-10-11T11:24:52.332 INFO:teuthology.orchestra.run.smithi111.stdout: 262320 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Oct 11 11:24 /home/ubuntu/cephtest 2023-10-11T11:24:52.334 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-10-11T11:24:52.367 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-10-11T11:24:52.390 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: CommandFailedError: Command failed on smithi111 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json' 2023-10-11T11:24:52.391 INFO:teuthology.nuke:Checking targets against current locks 2023-10-11T11:24:52.418 DEBUG:teuthology.nuke:shortname: smithi050 2023-10-11T11:24:52.419 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-11T11:24:52.450 DEBUG:teuthology.nuke:shortname: smithi111 2023-10-11T11:24:52.451 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-11T11:24:52.455 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi050.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/dis-2023-10-11_10:22:06-rbd-wip-dis-testing-distro-default-smithi/7421102', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-11 10:22:14.535993', 'locked_by': 'scheduled_dis@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='} 2023-10-11T11:24:52.475 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi111.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/dis-2023-10-11_10:22:06-rbd-wip-dis-testing-distro-default-smithi/7421102', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-11 10:22:14.537584', 'locked_by': 'scheduled_dis@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='} 2023-10-11T11:24:52.533 INFO:teuthology.orchestra.console.smithi050:Power off 2023-10-11T11:24:52.534 DEBUG:teuthology.orchestra.console.smithi050:pexpect command: ipmitool -H smithi050.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-10-11T11:24:52.635 INFO:teuthology.orchestra.console.smithi111:Power off 2023-10-11T11:24:52.636 DEBUG:teuthology.orchestra.console.smithi111:pexpect command: ipmitool -H smithi111.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-10-11T11:24:52.677 DEBUG:teuthology.orchestra.console.smithi050:power off output: Chassis Power Control: Down/Off 2023-10-11T11:24:52.678 DEBUG:teuthology.orchestra.console.smithi050:pexpect command: ipmitool -H smithi050.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-11T11:24:52.721 DEBUG:teuthology.orchestra.console.smithi111:power off output: Chassis Power Control: Down/Off 2023-10-11T11:24:52.722 DEBUG:teuthology.orchestra.console.smithi111:pexpect command: ipmitool -H smithi111.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-11T11:24:52.739 DEBUG:teuthology.orchestra.console.smithi050:check power output: Chassis Power is on 2023-10-11T11:24:52.767 DEBUG:teuthology.orchestra.console.smithi111:check power output: Chassis Power is on 2023-10-11T11:24:56.740 DEBUG:teuthology.orchestra.console.smithi050:pexpect command: ipmitool -H smithi050.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-11T11:24:56.768 DEBUG:teuthology.orchestra.console.smithi111:pexpect command: ipmitool -H smithi111.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-11T11:24:56.863 DEBUG:teuthology.orchestra.console.smithi050:check power output: Chassis Power is on 2023-10-11T11:24:56.881 DEBUG:teuthology.orchestra.console.smithi111:check power output: Chassis Power is on 2023-10-11T11:25:00.865 DEBUG:teuthology.orchestra.console.smithi050:pexpect command: ipmitool -H smithi050.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-11T11:25:00.888 DEBUG:teuthology.orchestra.console.smithi111:pexpect command: ipmitool -H smithi111.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-11T11:25:00.992 DEBUG:teuthology.orchestra.console.smithi050:check power output: Chassis Power is off 2023-10-11T11:25:01.023 DEBUG:teuthology.orchestra.console.smithi111:check power output: Chassis Power is off 2023-10-11T11:25:01.093 INFO:teuthology.orchestra.console.smithi050:Power off completed 2023-10-11T11:25:01.124 INFO:teuthology.orchestra.console.smithi111:Power off completed 2023-10-11T11:25:01.230 INFO:teuthology.run:Summary data: description: rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-stupid policy/none rbd-mirror/four-per-cluster supported-random-distro$/{rhel_8} workloads/rbd-mirror-snapshot-stress-workunit-fast-diff} duration: 3379.8909256458282 failure_reason: 'Command failed on smithi111 with status 124: ''sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json''' flavor: default owner: scheduled_dis@teuthology status: fail success: false 2023-10-11T11:25:01.230 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-10-11T11:25:01.311 INFO:teuthology.run:FAIL